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EXECUTIVE  SUMMARY 


Introduction 

The  demonstrated  two-phase  open-bath  immersion  (OBI)  cooling  technology  was  targeted  to 
substitute  for,  or  be  used  in  conjunction  with,  other  electronic  equipment  cooling  technologies  to 
significantly  reduce  the  electrical  energy  needed  for  high-performance  computing  (HPC)  data 
center  operation  across  the  U.S.  Department  of  Defense  (DoD). 

In  addition  to  the  electrical  energy  supplied  to  the  information  technology  (IT)  equipment  at 
HPC  sites,  a  significant  amount  of  electrical  energy  (cooling  energy)  is  required  to  remove  the 
heat  generated  by  the  IT  equipment.  In  fact,  energy  used  for  cooling  is  often  50  to  75  percent  of 
the  electrical  energy  supplied  to  the  electronic  equipment.  The  demonstrated  OBI  technology 
significantly  reduces  the  cooling  energy  by  immersing  the  electronic  equipment  in  a  bath  of 
dielectric  (non-conducting)  liquid. 

The  dielectric  liquid  used  for  this  demonstration  was  3M  Novec  649  Engineered  Fluid.  The  heat 
from  the  electronic  components  is  rejected  as  the  Novec  liquid  undergoes  a  phase  change  (liquid 
to  gas).  This  phase  change  takes  place  at  49°C,  so  relatively  warm  cooling  water  can  be  used  to 
condense  the  vapor  back  to  a  liquid.  A  warm-water  cooled  bath  is  more  energy  efficient  than 
typical  cooling  systems  that  use  much  cooler  water  from  compressor-based  systems.  The  water 
used  to  cool  two-phase  immersion-cooled  electronics  can  be  provided  by  simple,  economical 
“dry  coolers”  if  space  allows.  A  dry  cooler  is  a  water-to-air  heat  exchanger  that  includes  a  fan 
placed  in  the  outside  environment — very  similar  in  concept  to  an  automotive  radiator. 

This  demonstration,  which  took  place  at  the  U.S.  Naval  Research  Laboratory  (NRL)  in 
Washington  D.C.,  consisted  of  a  commercially  available  high-performance  computer  immersed 
in  the  3M  Novec  649  liquid.  The  immersion  cooling  system  was  tested  at  a  high  computer  load. 
Cooling  for  the  bath  was  provided  by  a  dry  cooler  located  outside  an  HPC  center  at  NRL. 

Summary  of  Performance  Objectives  and  Results 

The  demonstration  evaluated  twelve  performance  objectives.  The  performance  evaluations  were 
conducted  on  the  same  computer  system  being  cooled  with  a  standard  cooling  option  (Base 
Case)  and  with  the  demonstrated  immersion  cooling  technology.  Some  performance  objectives 
had  a  goal  and  a  “stretch”  goal.  The  goal  is  the  basic  performance  objective,  and  the  stretch  goal 
is  a  more  ambitious  objective. 

Some  efficiency-related  measurements,  planned  as  part  of  evaluating  certain  performance 
objectives,  were  not  available,  due  to  IT  equipment  failures.  Simulations  were  used  instead  to 
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provide  meaningful  results  for  the  affected  performance  objectives. 


POl:  Improved  Cooling  Energy  Efficiency 

The  cooling  energy  savings  objective  was  met.  The  savings  goal  was  50  percent,  and  the 
demonstration  resulted  in  72  percent  savings. 

PQ2:  Reduced  Overall  Data  Center  Site  Energy  Consumption 

Overall  site  energy  includes  the  energy  needed  by  the  IT  equipment,  data  center  infrastructure, 
and  all  energy  consuming  equipment  not  normally  thought  of  as  part  of  a  data  center  such  as 
generator  block  heaters  and  primary  power  distribution  losses. 

The  overall  data  center  energy  reduction  objective  was  met.  The  goal  was  a  reduction  of 
15  percent.  The  results  were  a  reduction  of  19  percent. 

PO  3:  Improved  Computational  Energy  Efficiency 

This  metric  measures  the  computing  accomplished  divided  by  the  electrical  energy  consumed  by 
IT  equipment. 

This  goal  was  not  met.  The  goal  was  better  or  equal  computational  efficiency  compared  to  the 
Base  Case.  The  Pilot  Test  (immersion  cooling)  had  809  MFLOPS/watt  and  the  Base  Case  (direct 
liquid  cooling)  had  857  MFLOPS/watt. 

Lower  energy  efficiency  for  the  Pilot  Test  (immersion  cooling)  is  likely  caused  by  the  higher 
CPU  temperatures  compared  to  the  Base  Case.  The  goal  is  not  likely  achievable  with  the  high 
boiling  temperature  of  Novec  649. 

PQ4:  Low  Concentrations  of  Novec  649  Vapors  During  Normal  Operation 
Novec  649  vapor  concentrations  were  measured  at  the  operator’s  breathing  zone  and  under  the 
floor  every  five  minutes  for  10  months.  Exposure  for  8-hour  time  weighted  average  (TWA) 
periods  were  evaluated.  The  TWA  maximum  for  Novec  649  vapor  is  <150  ppmV  (parts  per 
million  by  volume)  per  the  3M  Safety  Data  Sheet.  The  highest  8-hour  TWA  value  calculated  was 
48  ppmV.  Therefore,  the  goal  was  met. 

PQ5:  Low  Concentrations  of  Novec  649  Vapors  During  Startup  or  Maintenance 
The  vapor  concentration  limit  for  short  (less  than  4  continuous  hours)  exposure  periods  is 
<  100,000  ppmV  per  the  3M  Safety  Data  Sheet.  The  peak  concentration  measured  during  the 
demonstration  was  200  ppmV.  Therefore,  the  goal  was  met. 

PQ6:  Reduction  in  Direct  Greenhouse  Gas  Emissions 

The  carbon  dioxide  equivalent  emissions  (CCLe)  were  calculated  based  on  the  electrical  energy 


9 


savings  from  P02. 


The  goal  of  a  reduction  compared  to  the  Base  Case  was  met.  Simulations  estimated  a  carbon 
dioxide  equivalent  (CCBe)  emission  reduction  of  19  percent,  or  2,772  metric  tons  per  year  for  a 
simulated  data  center  designed  for  a  maximum  IT  equipment  load  of  2  megawatts  (MW). 

PQ7:  Dielectric  Liquid  Loss 

The  immersion  liquid  Novec  649  is  expensive  and  volatile  compared  to  other  cooling  fluids  (air 
and/or  water)  typically  used  for  data  centers.  The  metric  for  this  performance  objective  was  the 
cost  of  liquid  lost  divided  by  the  cost  of  electrical  energy  consumed  by  the  IT  equipment. 

This  goal  was  not  met.  The  cost  of  the  lost  liquid  was  368  percent  of  the  cost  of  the  IT  equipment 
energy  consumed,  compared  to  the  goal  of  1  percent.  Because  Novec  649  is  a  volatile  liquid  and 
its  vapor  is  invisible,  the  locations  of  vapor  or  liquid  leaks  were  not  evident.  There  will  be 
technical  challenges  containing  volatile  fluids.  Experiments  attempting  to  characterize  and 
isolate  the  fluid  loss  mechanisms  were  not  conducted. 

PQ8:  System  Economics 

A  simple  payback  period  and  optional  financial  net  present  value  (NPV)  analysis  for  a  seven- 
year  period  were  performed. 

The  payback  period  calculation  assumed  realistic  design  improvements  to  the  demonstrated 
technology;  most  importantly,  Novec  liquid  initial  fill  volume  and  the  cost  of  the  bath  enclosure. 
The  simple  payback  period  was  calculated  to  be  33  years,  therefore  neither  the  goal  (<  4  years) 
nor  the  stretch  goal  (<  3  years)  was  met.  The  immersion  cooling  option  had  a  9.5  percent  higher 
seven-year  NPV  than  the  Base  Case. 

The  initial  fill  volume  and  bath  cost  were  high  because  the  IT  equipment  used  in  the 
demonstration  was  not  specifically  designed  for  two-phase  immersion  cooling.  Before 
immersion  cooling  can  be  cost  competitive  with  existing  cooling  methods  there  needs  to  be  a 
substantial  increase  in  the  amount  of  IT  equipment  that  can  be  contained  in  a  given  volume.  This 
density  increase  may  involve  a  complete  rethinking  of  current  HPC  computing  architecture. 

PQ9:  Lower  CPU  Chip  Temperatures 

The  goal  of  the  central  processing  unit  (CPU)  temperature  for  the  demonstrated  technology  was 
to  be  equal  or  lower  than  the  Base  Case  temperature. 

This  goal  was  not  met.  The  CPU  temperatures  averaged  approximately  20°C  higher  when  the 
computer  was  immersion-cooled  compared  to  the  Base  Case.  This  higher  temperature  may  have 
been  due  to  a  couple  of  contributing  factors.  The  liquid  temperature  close  to  the  CPU  is  49C  in 
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the  immersion  case  (Novec  649  boils  at  49C)  and  20C  in  the  Base  Case  (20C  cooling  water). 
The  goal  is  not  likely  achievable  with  the  high  boiling  temperature  of  Novec  649.  The  other 
contributing  factor  is  that  the  phase  change  taking  place  on  chip  heat-transfer  surfaces  may  also 
have  deposited  pollutants,  which,  in  turn,  would  have  limited  the  heat  transfer. 

POIQ:  Higher  User  Satisfaction,  Low  Number  of  Concerns 

Personnel  at  the  demonstration  site  reported  on  safety  and  operational  concerns. 

The  goal  of  zero  unresolved  safety  concerns  was  met,  the  goal  of  zero  unresolved  operational 
concerns  was  not  met.  There  were  thirteen  (13)  unresolved  operational  concerns — most 
importantly,  IT  equipment  failures. 

Other  than  the  repeated  electronic  failures,  overcoming  the  remaining  operational  concerns  could 
also  be  a  major  technical  challenge. 

POll:  Improved  IT  Power  Density 

Equipment  floor  space  power  density  (in  kilowatts  per  square  foot,  kW/ft2)  was  estimated 
for  IT  equipment  cooled  using  the  immersion  technology  as  well  as  for  the  Base 
Case  technology. 

The  goal  of  a  higher  power  density  with  immersion  cooling  was  not  met.  The  demonstrated 
technology  had  a  power  density  of  just  22  percent  of  the  Base  Case.  An  important  factor  is  that 
the  baths  are  horizontal  and  are  not  able  to  use  space  for  electronics  much  above  three  feet; 
whereas,  conventional  racks  are  vertical  and  are  able  to  house  electronics  to  a  height  of  more 
than  six  feet.  To  achieve  a  comparable  density  to  the  Base  Case  would  be  a  major  technical 
challenge.  It  could  involve  a  complete  rethinking  of  current  HPC  computing  architecture  to 
significantly  increase  computational  density  in  the  bath 

PQ12:  System  Maintenance 

The  number  of  maintenance  requests  for  the  immersion-cooled  computer  equipment  was 
compared  to  field  data  from  installations  of  similar  computer  equipment  conventionally  cooled 
with  air. 

The  goal  for  this  performance  objective  was  not  met.  The  immersion-cooled  equipment  had  a 
6,643  percent  higher  service  request  rate  compared  to  the  Base  Case.  Repeated  logic  board  and 
power  supply  failures  were  primarily  responsible  for  the  high  number  of  service  requests. 

The  cause  of  the  power  supply  failures  was  determined  and  a  subsequent  fix  was  successfully 
applied.  Considerable  resources  were  assigned  to  find  and  correct  the  cause  or  causes  of  the 
logic -board  failures.  A  large  number  of  metallic  filaments  "tin  whiskers"  were  observed  on  failed 
boards.  Although  the  exact  mechanism  for  creating  these  tin  whiskers  is  unresolved,  they  likely 
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created  shorts  on  the  logic  boards.  Identifying  the  root  cause(s)  and  a  solution  for  the  logic  board 
failures  could  require  considerable  resources. 


Conclusions  and  Recommendations 

Two-phase  immersion  cooling  using  Novec  649  is  not  viable  at  this  time.  The  primary  obstacles 
that  need  to  be  overcome  are  IT  equipment  failures  and  costs.  However,  the  demonstrated 
technology  met  the  performance  objectives  for  energy  efficiency  and  greenhouse  gas  reduction. 
Before  commercialization  of  this  technology  can  occur,  a  root  cause  analysis  of  the  failures 
should  be  completed,  and  the  design  changes  proven. 


1.0  INTRODUCTION 


The  U.S.  Department  of  Defense’s  (DoD’s)  computational  needs  show  continual  growth, 
resulting  in  requirements  for  more  data  center  space  for  both  traditional  business  applications 
and  high-performance  computing  (HPC).  Electricity  use  for  these  data  centers  often  dominates 
the  electricity  demand  of  the  DoD  sites  where  they  operate.  The  DoD’s  Data  Center 
Consolidation  Plan  to  support  the  Federal  Data  Center  Consolidation  Initiative  will  aggressively 
consolidate  assets,  resulting  in  fewer  but  more  energy-intensive  facilities.  Consolidation,  growth 
in  DoD  high-performance  computing,  and  increasingly  energy-intensive  computing  systems 
drive  the  need  to  investigate  alternative  cooling  systems.  Traditional  air  cooling  of  electronic 
equipment  has  limitations  with  cooling  high-power  chips.  The  technology  demonstrated  in  this 
project  has  the  capacity  to  cool  extreme  heat  loads  very  effectively  without  using  precious  water 
resources. 


1.1  BACKGROUND 

Data  center  electrical  energy  use  for  powering  and  cooling  the  DoD’s  electronic  equipment  has 
increased  over  recent  decades,  driven  by  the  need  for  more  processing  capability.  It  is  all  but 
certain  that  the  need  for  computational  resources  will  continue  on  a  steep  upward  trajectory.  It  is 
important  to  reduce  the  energy  consumption  and  lower  the  carbon  dioxide  (CO2)  emissions  by 
improving  not  only  the  efficiency  of  the  electronic  equipment  but  also  that  of  the  cooling 
infrastructure. 

By  immersing  the  IT  equipment  in  a  non-conductive  (dielectric)  liquid — 3M  Novec™  649 
Engineered  Fluid  (649  liquid) — the  technology  demonstrated  reduces  the  energy  needed  to  cool 
HPC  electronics.  This  liquid  provides  cooling  by  changing  phase  (liquid-to-gas)  at  the  surface  of 
hot  electronic  equipment  components.  This  particular  liquid  boils  at  49°C,  and  the  process  of 
boiling  has  the  capacity  to  remove  very  high  heat  loads. 

The  649  liquid  is  working  well  in  experimental  setups  at  the  Mayo  Clinic  (Polzer,  pers.  comm. 
2015).  There  have  been  reports  that  the  chemical  was  sensitive  to  liquid  water;  however,  it  may 
still  be  considered  a  promising  technology.  This  project  was  initiated  to  establish  its  commercial 
applicability. 

Appendix  A  lists  the  project  Points  of  Contact. 


1.2  OBJECTIVE  OF  THE  DEMONSTRATION 

The  primary  project  objective  was  to  demonstrate  efficient  cooling  of  high-heat  density 
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electronics  by  the  use  of  two-phase  open-bath  immersion  (OBI)  cooling.  The  demonstration  was 
conducted  over  a  considerable  time  period  in  order  to  identify  potentially  undesirable  operational 
issues.  This  demonstration  was  on  a  larger  scale  than  previous  proof-of-concept  demonstrations 
and  was  meant  to  demonstrate  the  viability  of  this  technology  at  commercial  scale. 

Open-bath  immersion  cooling  can  efficiently  cool  high-density  electronics  in  data  centers 
without  the  need  for  compressor-based  cooling.  Since  this  system  operates  well  using  high- 
temperature  coolant,  dry  coolers  can  be  used  for  heat  rejection  to  the  atmosphere,  thereby 
eliminating  evaporative  water  use  almost  anywhere  in  the  world. 

In  addition  to  performance  objective  evaluations  related  to  energy  efficiency,  a  number  of  other 
objectives  were  evaluated  during  the  demonstration,  including  computer  equipment  floor  space 
power  density,  electronic  component  temperatures,  safety  (chemical  exposure),  and  the  cost  of 
dielectric  liquid.  In  an  attempt  to  identify  short-  to  medium-term  reliability  issues,  the 
demonstration  was  operated  for  approximately  10  months,  running  exercising  software  when 
production  software  loads  were  not  applied. 

Liquid  immersion  cooling,  especially  with  phase  change,  is  a  paradigm  shift  in  the  way 
electronics  are  cooled.  The  demonstration  was  meant  as  an  important  step  in  introducing  the 
technology  to  a  broader  audience  through  technology  transfer  to  the  DoD  and  other  stakeholders. 
The  U.S.  Department  of  Energy’s  (DOE’s)  Federal  Energy  Management  Program  (FEMP)  has 
designated  Lawrence  Berkeley  National  Laboratory  (LBNL)  as  the  “Center  of  Expertise  for 
Energy  Efficiency  in  Data  Centers.”  The  Center  is  able  to  widely  disseminate  the  results  of  the 
demonstration  to  the  DoD,  the  Federal  sector  in  general,  and  industry  at  large. 

1.3  REGULATORY  DRIVERS 

A  number  of  regulatory  drivers  have  spurred  the  need  for  this  kind  of  technology: 

Executive  Order  (EO)  13423  “Strengthening  Federal  Environmental,  Energy,  and 
Transportation  Management” 

Energy  use  in  DoD  data  centers,  if  separately  metered,  is  exempt  from  energy  reduction  goals, 
but  not  water  reduction  goals.  The  water  reduction  goal  of  16  percent  by  the  end  of  FY  2015 
compared  to  FY  2007  will  be  challenging  to  meet.  The  demonstrated  immersion  cooling 
technology  has  been  purported  to  eliminate  water  use. 

Executive  Order  (EO)  13514  “Federal  Leadership  in  Environmental,  Energy,  and 
Economic  Performance” 

The  sustainability  goal  of  achieving  net  zero  energy  use  by  FY  2030  (start  of  design  in  2020  or 
later)  requires  aggressive  energy  efficiency  strategies,  and  data  centers  are  not  exempted.  In 
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addition,  because  the  system  will  operate  with  elevated  temperatures,  there  may  be  an 
opportunity  to  reuse  the  heat  produced  by  the  electronics.  This  technology  may  also  help  meet 
the  water  reduction  goals  of  20  percent  reduction  by  FY  2020  compared  to  FY  2010  by 
eliminating  all  water  use  for  evaporative  cooling,  as  well  as  meeting  the  product  and  stewardship 
goal  of  improving  water  efficiency. 

Executive  Order  (EO)  13693  “Planning  for  Federal  Sustainability  in  the  Next  Decade” 

The  demonstrated  technology  may  assist  Federal  data  centers  in  maximizing  energy  efficiency 
and  water  use  of  “core”  data  centers.  Executive  Order  section  3(a)(ii)  instructs  the  head  of  each 
agency  to  improve  data  center  energy  efficiency  at  agency  facilities  by: 

(A)  ensuring  the  agency  chief  information  officer  promotes  data  center  energy 
optimization,  efficiency,  and  performance; 

(B)  installing  and  monitoring  advanced  energy  meters  in  all  data  centers  by  fiscal 
year  2018; and 

(C)  establishing  a  power  usage  effectiveness  target  of  1.2  to  1.4  for  new  data 
centers  and  less  than  1.5  for  existing  data  centers. 

The  demonstrated  technology  has  the  capability  to  meet  requirement  C  with  a  significant  margin. 

Industry  Guidelines 

The  demonstrated  technology  can  utilize  the  liquid  cooling  thermal  guidelines  developed  by 
ASHRAE  Technical  Committee  9.9.  Mission  Critical  Facilities,  Data  Centers,  Technology 
Spaces,  and  Electronic  Equipment.  These  guidelines  are  in  the  ASHRAE  Datacom  book  series, 
both  in  Thermal  Guidelines  for  Data  Processing  Environments  and  Liquid  Cooling  Guidelines 
for  Datacom  Equipment  Centers.  Higher  temperatures  for  liquid  cooling  as  defined  in  the 
guidelines  allow  cooling  with  dry  coolers  in  most  climates,  thereby  eliminating  water  usage. 

Energy  Policy  Act  of  2005 

The  Energy  Policy  Act  of  2005  requires  that  Federal  buildings’  energy  performance  exceed 
ASHRAE  Standard  90.1  by  30  percent.  Previously,  data  center  facilities  were  excluded  from  this 
requirement;  however,  the  exclusion  has  been  removed  in  the  current  ASHRAE  Standard.  It  may 
be  extremely  difficult  to  achieve  a  30  percent  energy  saving  without  a  disruptive  technology  such 
as  liquid  immersion  cooling.  In  addition,  immersion  cooling  may  achieve  energy  reduction  in  the 
FT  equipment  (e.g.,  fan  removal),  as  well  as  in  the  heat  rejection  path. 

U.S.  Navy  Policy  OPNAY  4100.5E 

OPNAV  Instruction  4100. 5E  spells  out  the  Navy  policy  for  Navy  shore  energy.  This  includes 
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technological  innovation  to  enable  commands  to  meet  the  Navy’s  land-based  energy  goals.  It 
also  requires  commands  to  submit  energy  reduction  plans  annually. 


2.0  TECHNOLOGY  DESCRIPTION 
2.1  TECHNOLOGY  OVERVIEW 
Technology  Theory,  Functionality,  and  Operation 

Servers  were  immersed  in  modular  baths  containing  a  dielectric  (not  electrically  conductive) 
liquid.  This  demonstration  used  3M’s  fluoroketone  (FK)  Novec  649,  chemical  formula 
CF3CF2C(0)CF(CF3)2.  This  project  used  a  semi-open  immersion  cooling  technology.  The  term 
semi-open  denotes  a  bath  with  a  lid,  which  is  open  only  when  access  is  needed.  Although  the 
bath  in  this  demonstration  had  a  semi-open  lid  design  (it  was  removed  for  FT  equipment  service), 
this  report  refers  to  it  as  open-bath  immersion  (OBI)  cooling. 

Electronic  components  are  cooled  by  convection  or  when  the  Novec  649  liquid  boils  (changing 
phases  from  a  liquid  to  a  gas,  which  occurs  at  49°C)  near  high-heat  generating  components.  The 
gas,  which  is  less  dense  than  the  liquid  but  denser  than  air,  rises  to  the  space  above  the  liquid, 
where  it  comes  into  contact  with  a  condenser  integrated  into  the  bath  (Figure  2-1).  The  vapor  is 
condensed  back  to  a  liquid  by  the  condenser,  which  is  cooled  by  a  water  loop  connected  to  a 
source  of  cooling  water.  The  condensate  falls  as  droplets  back  into  the  liquid  (Figure  2-1). 
Recirculation  or  return  pumps  are  not  needed  for  either  phase  (liquid  or  gas)  for  Novec  649  two- 
phase  immersion  cooling.  The  vapor  generated  in  the  boiling  process  forms  a  distinct  layer 
above,  which  is  a  region  of  air  and  vapor  called  the  headspace. 

The  heat  output  from  the  servers  changes  with  the  work  load.  This,  in  turn,  causes  the  vapor  level 
to  change.  The  thermal  control  keeps  the  vapor  level  within  desired  limits  by  modulating  the 
cooling  water  flow  rate. 

Immersed  IT  equipment  can  be  removed  for  service  by  opening  the  lid  and  simply  lifting  the 
equipment  out  of  the  tank.  When  servers  are  removed  slowly  from  the  bath  through  the  vapor, 
liquid  on  the  equipment  surfaces  quickly  evaporates  and  is  captured  by  the  condenser.  Thus,  the 
servers  leave  the  bath  essentially  dry,  causing  minimal  liquid  loss  due  to  normal  maintenance. 

The  OBI  system  operates  at  atmospheric  pressure,  and  electrical  connections  enter  the  bath  from 
above,  through  a  sealed  conduit  that  terminates  beneath  the  liquid  level. 
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Technology  Overall  Schematic 

Cooling  water  flows  through  the  condensing  coil,  which  removes  heat  from  the  vapor.  Since 
Novec  649  (Figure  2-1,  “Novec  Liquid”)  boils  at  49°C,  the  temperature  of  the  cooling  water 
(Figure  2-1,  “Cooling  Water”)  can  be  significantly  higher  than  that  typically  found  in  HPC  data 
centers.  The  system  studied  may  work  with  40°C-45°C  (104°F-113°F)  cooling  water  compared 
with  7°C-20°C  (45°F-68°F)  for  other  cooling  technologies.  These  higher  temperatures  (40°C- 
45°C  [104°F-113°F])  can  generally  be  produced  without  mechanical  (compressor-based) 
refrigeration. 

The  heat  removed  by  the  cooling  water  is  rejected  to  the  outside  atmosphere  using  a  “dry 
cooler” — a  water-to-air  heat  exchanger  similar  to  an  automobile  radiator.  The  dry  cooler  has  a 
fan  that  boosts  the  heat  transfer  when  needed.  Figure  2-2  shows  the  fan,  pump,  and  valve  layout 
used  in  the  demonstration.  The  pump  and  fan  require  energy  to  move  the  cooling  water  and 
modulate  the  temperature  of  the  cooling  water  supplied  by  the  dry  cooler. 


5 


Other  cooling  water  infrastructure  options  are  possible,  e.g.,  using  chilled  water  already  available 
in  the  building  or  adding  a  dedicated  cooling  tower.  The  project  team  selected  a  dry  cooler 
option  to  demonstrate  that  a  low  first-  and  operational-cost  cooling  option  is  feasible. 


Figure  2-2:  Demonstration  Cooling  System  Schematic 


2.2  TECHNOLOGY  DEVELOPMENT 

Improving  the  energy  efficiency  associated  with  cooling  IT  equipment  as  well  as  improving  the 
density  limitations  of  air-cooled  IT  equipment  has  been  a  focus  in  recent  years.  Air-cooled 
equipment  has  traditionally  been  cooled  by  using  room-level  air  conditioning  systems.  Each 
piece  of  IT  air-cooled  equipment  typically  has  its  own  fans  that  pull  in  cool  room  air  at  the  front 
and  exhausts  hot  air  to  the  rear.  This  hot  air  is  returned  to  the  room  air  conditioning  system  to  be 
cooled  again  often  using  cooling  towers  and  chillers  to  reject  the  heat  outside.  As  IT  equipment 
heat  density  continues  to  increase,  two  issues  are  encountered:  (1)  Legacy  data  center  air 
conditioning  is  not  able  to  supply  enough  cool  air,  and  (2)  IT  equipment  that  uses  high-power 
CPU  components  face  significant  cooling  design  challenges  if  restricted  to  using  only  air 
for  cooling. 

Alternate  cooling  methods  have  been  introduced  in  the  last  decade  in  an  attempt  to  address  the 
limitations  of  air-cooled  IT  equipment  and  find  better  efficiency.  One  approach  is  to  enclose  the 
IT  equipment  at  the  rack  level.  Rack-level  cooling  can  provide  efficiency  gains  as  outlined  in 
Demonstration  of  Rack-Mounted  Computer  Equipment  Cooling  Solutions  (Coles  2014).  Another 
more  recent  approach  is  to  bring  water  directly  to  the  chip,  using  a  small  cold  plate  (direct- 
cooling)  which  is  not  a  new  concept  but  is  being  popularized  by  solutions  that  can  be  adapted  to 
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standard  IT  equipment.  The  demonstrated  results  of  one  company’s  direct-cooling  solution  that 
improved  overall  data  center  efficiency  is  presented  in  Direct  Liquid  Cooling  for  Electronic 
Equipment  (Coles  and  Greenberg  2014).  Another  more  recent  method  involves  immersing  the  IT 
equipment  in  a  liquid  that  does  not  change  phase.  This  method  is  termed  single-phase  immersion 
cooling.  At  least  two  liquid  types  are  currently  being  offered  for  this  type  of  immersion  cooling. 
Two  such  examples  of  companies  offering  single -phase  cooling,  each  using  a  different  liquid,  are 
LiquidCool  Solutions  and  Isotope  (using  Novec  HFE  liquid)  and  Green  Revolution  Cooling 
(using  mineral  oil). 

In  the  quest  for  more-efficient  and  higher-performance  heat  transfer,  two-phase  immersion 
cooling  technologies  are  being  developed  using,  for  example,  Novec  649,  a  fluoroketone  (FK) 
liquid,  and  Novec  7100,  a  hydrofluoroether  (HFE)  liquid.  The  liquid  used  in  this  demonstration 
was  Novec  649,  which  is  an  FK  liquid  with  a  low  global  warming  potential  (GWP)  of  1 
(SDS  2015a). 

Hydrofluoroether  liquids  generally  have  higher  GWP  ratings;  for  example,  Novec  HFE-7100  has 
a  GWP  rating  of  320  according  to  its  Safety  Data  Sheet  (SDS)  (SDS  2015b).  They  also  are 
reported  to  create  a  dielectric  environment  that  may  pose  limitations  with  the  high-speed 
circuitry  in  the  IT  equipment  used  in  this  demonstration  (Chan  et  al.  2010).  Therefore,  the  Novec 
649  was  selected  for  its  superior  dielectric  characteristics  and  low  GWP. 

A  publically  documented  track  record  of  using  Novec  liquids  for  two-phase  immersion  cooling 
of  IT  electronics  is  limited.  However  a  number  of  companies  and  individuals  have  worked  on 
developing  two-phase  cooling,  and  in  some  cases  are  offering  cooling  solutions  using  Novec  FK 
and  HFE  liquids.  The  following  examples  are  some  applications  of  Novec  Engineered  Fluids  for 
two-phase  immersion  cooling: 

Mayo  Clinic 

The  Special  Purpose  Processor  Development  Group  (SPPDG)  within  the  Mayo  Clinic 
was  introduced  to  3M’s  immersion  cooling  thermal  management  approach  via  a 
presentation,  The  Merits  of  Open  Bath  Immersion  Cooling  of  Datacom  Equipment  (Tuma 
2010),  at  SEMI-THERM  2010. 

The  SPPDG  completed  experiments  to  characterize  the  optical  and  electrical  signal 
integrity  performance  across  a  couple  of  different  dielectric  liquids;  Novec  649  was 
included.  The  results  were  presented  at  IMAPS-ATW  2010  (Chan  et  al.  2010).  The 
results  concluded  that  an  FK  fluid  (Novec  649)  appears  able  to  maintain  electrical  signal 
integrity  exceeding  15  gigahertz  (GHz)  and  HFE  fluids  may  be  unable  to  maintain 
electrical  signal  integrity  above  a  few  GHz. 
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The  SPPDG  assembled  a  small  form  factor  demonstration  test  containing  a  Dell  OptiPlex 
desktop  computer)  to  investigate  potential  idiosyncrasies  that  might  be  associated  with 
this  (two-phase)  thermal  management  solution.  The  test  vehicle  started  running  in  the 
early-to-mid  December  2010  time  frame.  The  SPPDG  did  run  into  a  few  hurdles,  related 
to  water  contamination  and  deposits  of  oil  from  cable  materials,  along  the  way.  These 
hurdles  were  documented  in  Design  Considerations  Relating  to  Non-Thermal  Aspects  of 
Passive  2 -Phase  Immersion  Cooling  (Tuma  2011). 

Note:  Perfluoropropionic  acid  (PFPA)  can  form  when  Novec  649  comes  into  contact  with 
water.  Dioctyl  phthalate  (DOP)  is  used  as  a  primary  plasticizer  in  cables  made  from 
polyvinyl  chloride  (PVC).  The  PFPA  (acid)  and  DOP  (“goop”)  can  concentrate  in  areas 
where  Novec  649  boils,  and  are  suspected  of  causing  reliability  issues. 

The  Dell  computer  at  SPPDG  has  been  exposed  to  PFPA  and  DOP,  starting  in  December 
2010.  It  has  been  exposed  24  hours  per  day,  seven  days  a  week,  with  a  100  percent  CPU 
load  for  almost  five  years  and  is  still  running  at  the  time  of  this  report. 

In  addition,  SPPDG  tested  the  electrical  properties  of  the  liquid  after  two  years  of  use  and 
did  not  see  a  change  in  the  electrical  signaling  performance  characteristics. 

In  2011  SPPDG  built  a  large  test  tank  to  evaluate  how  the  performance  of  this  thermal 
management  platform  would  scale.  Experiments  from  this  effort  showed  that  the 
condenser  coils  performed  as  expected.  The  electric  cartridge  heaters  that  supplied  the 
load  in  this  experiment  had  been  incorrectly  sized  for  the  application.  This  eventually 
caused  a  catastrophic  arc  failure  to  occur  on  one  of  the  cartridge  heaters.  Arc  events  in 
liquid  Novec  649  have  the  potential  of  forming  perfluoroisobutylene  (PFIB),  which  is  a 
hazardous  chemical.  After  the  heating  rod  failure,  the  liquid  was  tested  at  3M  and  came 
back  negative  for  PFIB.  The  SPPDG  followed  up  by  testing  the  electrical  properties  of 
the  liquid  and  found  the  dielectric  environment  properties  of  the  liquid  were  unchanged 
as  well  (Polzer,  pers.  comm.  2015).  The  SPPDG  presented  the  results  from  this  effort  (not 
including  the  heater  cartridge  failure)  at  the  IMAPS  Advanced  Thermal  Workshop  2012 
(Polzer  and  Gilbert.  2012). 

Bitcoin  Mining  Electronics  Cooling  Development 

In  January  2013,  inventor/consultant  Mark  Miyoshi  began  development  of  a  two-phase 
cooling  system  using  Novec  649  to  be  used  for  cooling  bitcoin  mining  hardware.  After  a 
short  trial  period,  hardware  power  supply  and  logic -board  failures  occurred.  Novec  7100 
was  substituted  for  Novec  649  in  August  2014,  and  failures  ceased  (Miyoshi,  personal 
communication  2015). 
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Allied  Control 

In  November  2013  Allied  Control,  an  engineering  company  specializing  in  two-phase 
immersion  cooling,  announced  a  500  kW  installation.  Allied  Controls  claimed  that  the 
technology  is  universal  and  will  work  with  any  hardware,  including  CPU-based  systems 
that  use  Intel  processors  (press  release).  The  liquid  used  was  Novec  7100,  an  HFE  that 
boils  at  61°C  (Allied  Control  2013).  Alex  Kampl  from  Allied  Control  explains  that,  from 
a  cost  perspective,  two-phase  immersion  cooling  requires  a  hardware  density  much 
higher  than  typical  air-cooled  hardware.  Kampl  explains  the  design  features  of  their 
immersion  technology  that  uses  a  Novec  HFE  (Kampl  2014). 

Allied  Control  also  offers  two-phase  cooling  solutions  using  either  Novec  7100  or  Novec 
649  (Allied  Control  2015). 

EXTOLL 

A  German  company,  EXTOLL,  makes  high-performance  interconnection  technology  and 
offers  an  electronic  equipment  cooling  cabinet  called  GreenICE.  This  is  a  two-phase 
cooling  solution  that  uses  Novec  649  (EXTOLL  2015). 

Expected  Applications 

Industry  has  been  researching  and  developing  single -phase  immersion  liquid-cooled  solutions  for 
military  applications  for  years.  Dielectric  (non-conducting)  liquids  have  been  used  for  cooling 
avionics  in  military  applications  for  decades  (e.g.,  Skybolt  missiles  in  the  1960s)  and  have  been 
proposed  for  cooling  electronic  equipment  on  the  ground  (LiquidCool  Solutions  using  a  3M 
pumped  single-phase  liquid)  at  forward  operating  bases  supporting  combat  forces. 


A  promising  area  of  application  for  the  demonstrated  two-phase  cooling  technology  could  be 
high-density  electronic  equipment  found  in  many  data  centers.  In  the  current  demonstration,  the 
electronic  equipment  was  of  the  type  found  in  high-performance  computing  data  centers. 

2.3  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

Two  key  aspects  must  be  considered  when  comparing  the  demonstrated  technology  to  alternate 
existing  cooling  technologies:  (1)  their  ability  to  provide  high  energy  efficiency,  and  (2)  their 
ability  to  cool  high-heat  density  equipment.  Lor  example,  free-air  cooling  (use  of  free  cool 
outdoor  air  when  conditions  are  favorable)  is  energy  efficient  but  may  not  be  able  to  provide  the 
cooling  required  by  some  HPC  equipment. 

A  number  of  alternative  available  cooling  technologies  should  be  considered,  along  with  the  two- 
phase  immersion  technology  used  in  this  demonstration. 

Mineral  Oil  Immersion  Cooling 

In  the  past  few  years,  an  alternative  immersion  technology  using  mineral  oil  has  been 
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successfully  demonstrated  and  recently  deployed  by  Green  Revolution  Cooling  in  commercial 
data  centers.  This  alternative  uses  what  is  termed  a  white  mineral  oil  rather  than  the  3M  Novec 
649  Engineered  Fluid.  One  significant  advantage  of  the  mineral  oil  is  its  low  cost  compared  to 
the  3M  fluid.  However,  a  limitation  with  mineral  oil  cooling  is  the  maximum  heat  density  it  can 
support,  which  is  considerably  lower  compared  to  Novec  649.  Intel  processor-based  data  center 
equipment  is  typically  designed  to  be  air  cooled,  and  is  not  of  the  density  required  to  take  full 
advantage  of  the  Novec  649  two-phase  cooling  capability.  Therefore,  at  the  time  of  this  report, 
the  lower-heat  density  capability  of  mineral  oil  immersion  cooling  may  not  be  significant. 

Since  both  immersion  mineral  oil  and  Novec  649  cooling  designs  restrict  service  to  vertical 
removal  of  IT  equipment,  the  floor-space  density  of  immersion-cooled  IT  equipment  may  be 
limited  compared  to  typical  data  center  air-cooled  equipment  housed  in  tall  racks.  This  floor- 
space  density  limitation  may  pose  a  significant  drawback  associated  with  replacing  air-cooled  IT 
equipment  with  immersion-cooled  equipment.  There  are  reports  of  bitcoin  mining  companies 
vertically  stacking  two-phase  immersion  baths  to  improve  the  floor  space  density,  but  this 
approach  is  likely  not  practical  or  possible  in  legacy  data  centers. 

Free- Air  Cooling 

As  mentioned  above,  another  alternative  cooling  technology  has  been  named  free-air  cooling. 
Facebook  increased  the  awareness  of  this  cooling  method  by  publicizing  the  design  and 
efficiency  of  their  Prineville,  Oregon,  data  center.  A  free-air  cooled  data  center  introduces  air 
directly  from  the  outside  and  uses  this  air  to  cool  the  IT  equipment,  thus  avoiding  the  need  to  use 
cooling  towers  and  chillers  to  condition  the  data  center.  Free-air  cooled  data  centers  need  to  be 
located  in  areas  with  suitable  temperature  and  humidity  conditions.  Disadvantages  of  free-air 
cooling  include  control  issues  during  rapid  weather  changes  and  risks  of  contamination  from 
wildfires  or  dust  storms.  Also,  free-air  cooled  data  centers  may  be  viewed  as  more  vulnerable  to 
direct  physical  attacks  (security  issues).  The  IT  equipment  specifically  designed  to  take 
advantage  of  free-air  cooling  may  actually  be  less  expensive  relative  to  popular  commercial 
servers.  Information  regarding  the  design  of  IT  equipment  optimized  for  free-air  cooling  is 
available  for  free  via  the  Open-Compute  Project  founded  by  Facebook.  Free-air  cooling  is  not 
free  from  a  cost  perspective.  Fan  systems  are  needed  to  help  move  air  through  the  data  center, 
and  humidity  control  systems  may  be  needed. 

Hybrid  Cooling 

Hybrid  cooling  is  a  combination  of  direct  cooling  (water  to  cold  plates  on  each  processor)  and  air 
cooling.  The  air  cooling  part  gathers  heat  not  captured  by  the  direct  cooling.  Hybrid  cooling  is 
usually  enclosed  in  a  custom  cabinet  that  minimizes  the  cooling  load  on  the  data  center  room  air 
conditioning  system  by  capturing  almost  all  heat  and  transferring  the  heat  to  the  building  water 
loop.  As  mentioned  in  Section  5.1,  the  IT  equipment  for  the  Base  Case  was  cooled  by  a 
commercially  available  Silicon  Graphics  International  Corp.  (SGI)  hybrid  cooling  system. 
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Hybrid  cooling  systems  have  an  advantage  over  air  cooled  systems  since  much  of  the  heat  is 
transferred  at  high  temperatures  to  the  building  cooling  water  loop,  and  therefore  the  relatively 
inefficient  room  air  conditioning  requirements  are  lower.  Hybrid  cooling  can  also  take  the  less- 
integrated  form  of  modifying  existing  air-cooled  IT  equipment  to  include  cold  plates  usually 
cooled  by  water  attached  to  each  processor.  The  servers  are  modified,  at  an  extra  cost,  and  placed 
in  a  standard  type  data  center  server  rack  that  has  been  modified  to  provide  the  cooling  water 
tubing  needed  for  each  server.  The  additional  hardware  and  systems  needed  as  part  of  the  direct 
cooling  system  is  generally  thought  as  an  additional  cost.  The  advantage  of  hybrid  cooling  is  the 
ability  to  cool  high  heat  density  IT  equipment  without  imposing  a  significant  additional  load  on 
the  data  center  room  air  conditioning  system,  which  can  be  prohibitively  costly  to  modify. 

Enclosed-Rack  Air  Cooling 

Enclosed-rack  air  cooling  takes  the  form  of  housing  air-cooled  IT  equipment  in  a  single  or 
sometimes  multi-rack  arrangement.  Fans  that  are  part  of  the  rack  enclosure  move  hot  air  from  the 
back  of  the  servers  through  an  air-to-water  heat  exchanger  and  then  bring  the  cooled  air  back  to 
the  server  air  inlet  area.  Enclosed-rack  air  cooling  may  require  additional  floor  space. 

Two-Phase  3M  Novec  649  Immersion  Cooling 

Two-phase  immersion  cooling  should  have  distinct  energy  efficiency  and  local  heat  density 
advantages  compared  to  most  other  non-immersion  cooling  methods.  Novec  649,  used  in  this 
demonstration,  boils  and  therefore  condenses  at  49  °C.  The  water  needed  to  condense  the  vapor 
can  therefore  be  supplied  all  year  in  many  climates  using,  for  example,  a  dry  cooler  rather  than 
the  cooling  tower  and  chiller  found  at  many  data  centers.  Two-phase  immersion  cooling  has 
proven  to  cool  very  dense  heat  loads;  however,  the  current  configuration  of  commercial  IT 
equipment  is  not  designed  to  take  full  advantage  of  the  high-heat  density  capability.  Other 
advantages  include  reduced  noise  inside  the  data  center  and  the  potential  for  low  cooling 
infrastructure  costs. 

Two-phase  immersion  liquids  are  expensive.  Novec  649  is  $75/liter  ($284/U.S.  gallon).  Initially 
filling  a  two-phase  immersion  bath  will  cost  a  considerable  amount.  The  cost  to  fill  the  bath  for 
this  demonstration  was  approximately  $44,625.  Immersion  cooling  baths  are  somewhat  limited 
in  terms  of  floor-space  density  because  they  are  constructed  in  a  horizontal  orientation  compared 
to  the  usual  vertical  data  center  rack. 

The  construction  of  the  bath  and  attached  support  systems  (including  filtering,  spill  containment, 
venting  controls,  and  lid  sealing  devices)  tend  to  raise  the  bath  capital  cost  for  a  given  amount  of 
IT  equipment  compared  to  the  simple  and  inexpensive  data  center  rack.  The  additional  cost  to 
contain  the  IT  equipment  in  the  data  center  may  be  offset  by  a  low  capital  cost  of,  for  example, 
cooling  towers  and  chillers. 
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3.0  PERFORMANCE  OBJECTIVES 
3.1  PERFORMANCE  OBJECTIVE  RESULTS 

Table  3.1  lists  the  performance  objective  results  by  name  and  reference  number  in  the 
Demonstration  Plan  v  1.6,  dated  January  28,  2013,  for  this  ESTCP  project  (EW-201347). 


Table  3-1:  Performance  Objective  Results 


Performance 

Objective 

Metric 

Data  Requirements 

Success  Criteria 

Results 

Quantitative  Performance  Objectives 

1:  Improved  Cooling 
Energy  Efficiency 

pPUE  =  (IT  energy  + 
cooling  systems 
support  energy)  -f  IT 
energy 

climate  and  data 
center  assumptions, 
simulation  results 

50%  lower  than 
the 

Base  Case 

Goal:  Met 

73%  lower  than  the 
Base  Case 

2:  Reduced  Overall 
Data  Center  Site 
Energy 
Consumption 

yearly  energy 
consumption 
(kilowatts 
[kWh]/year) 

climate  and  data 
center  assumptions, 
simulation  results 

Goal:  15%  lower 
than  the  Base 
Case 

Stretch  Goal: 
20%  lower  than 
the  Base  Case 

Goal:  Met 

19%  lower  than  the 
Base  Case 

Stretch  Goal: 

Not  met 

3:  Improved 
Computational 
Energy  Efficiency 

compute  rate  H- 
IT  power 

compute 

performance, 

IT  power 
consumption 

Goal:  better  or 
equal  to  the  Base 
Case 

Stretch  Goal: 
10% 

improvement 

Goal  and  Stretch 
Goal:  Not  met 

5.6%  less  than  the 
Base  Case 

4:  Low 

Concentration  of 
Novec  649  Vapors 
During  Normal 
Operation 

8-hour  time-weighted 
average  (TWA)  of 
Novec  649  vapor 
measurements 

vapor  concentration 
level  measurements 

Goal:  zero 
periods  above 
150  ppmV 
Stretch  Goal: 
zero  periods 
above  75  ppmV 

Goal  and  Stretch 
Goal:  Met 

maximum  observed 
TWA  concentration 
was  48  ppmV 

5:  Low 

Concentration  of 
Novec  649  Vapors 
During  Startup  or 
Maintenance 

number  of  events  with 
Novec  649  vapor 
levels  above  no 
observed  adverse 
effect  limit  (NOAEL) 
of  100,000  pprnV 

vapor  concentration 
level  measurements 

zero  periods 
above 

100,000  ppmV 

Goal:  Met 

highest  observed 
concentration  = 

200  ppmV 
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Table  3-1:  Performance  Objective  Results  (continued) 


Performance 

Objective 

Metric 

Data  Requirements 

Success  Criteria 

Results 

6:  Reduction  in 
Direct  Greenhouse 
Gas  Emissions 

Carbon  dioxide 
equivalent  (CO^e) 
(metric  tons) 

Performance 
Objective  1  and  2 
simulation  results 

less  than  Base 
Case 

Goal:  Met 

19%  Reduction 

7:  Dielectric  Liquid 
Loss 

cost($)  liquid  loss  H- 
cost($)  IT  energy 

energy  used, 
liquid  lost 

Goal:  1% 

Stretch  Goal: 
0.1% 

Goal:  Not  met 
Liquid  Loss  360% 

Stretch  Goal: 

Not  met 

8:  System 
Economics 

savings  estimated 
from  National 
Institute  of  Standards 
and  Technology 
(NIST)  building  life 
cycle  cost  (BLCC) 
analysis 

simulated  energy 
use, 

estimated  and 
observed 
equipment  and 
liquid  costs 

Goal: 

4-year  payback 

Stretch  Goal: 
3-year  payback 

Goal  and  Stretch 
Goal:  Not  met, 
payback  33  years 

Net  present  value 
analysis  indicates 
immersion  cooling 
option  is  not 
recommended 

9:  Lower  CPU  Chip 
Temperatures 

compare  temperatures 
Base  Case  vs. 
Immersion-Cooled 
Case 

CPU  chip 
temperature 

measurements 

at  or  below 
Base  Case 

Goal:  Not  met, 

CPU  temperatures 
were  20°C  higher 
than  Base  Case 

Qualitative  Performance  Objectives 

10:  High  User 
Satisfaction,  Low 
Number  of  Concerns 

number  of  safety 
concerns; 

number  of  unresolved 
safety  concerns; 
number  of  operational 
concerns; 

number  of  unresolved 
operational  concerns 

demonstration  staff 
provides  thoughts 
on  safety  and 
operational  concerns 

Goal: 

zero  unresolved 
safety  concerns 

Stretch  Goal: 
zero  unresolved 
operational 
concerns 

Goal:  Met, 
zero  unresolved 
safety  concerns 

Stretch  Goal: 

Not  met, 

13  unresolved 
operational  concerns 

1 1 :  Improved  IT 
Power  Density 

Density; 

IT  (kW)  /  floor 
space  (ft2) 

high  density  layout 
using  Base  Case 
equipment,  layout 
using  demonstration 
equipment 

higher  than  the 
Base  Case 
configured  as  a 
typical  HPC 
installation 

Goal:  Not  met 

Immersion  Case 

22%  of  the 

Base  Case 

12:  System 
Maintenance 

number  of 

maintenance  requests 
for  immersed  IT 
compared  to 
traditionally  cooled 
SGI  equipment 

maintenance  request 
history  for 
immersed 
equipment,  typical 
maintenance  history 

immersed 
equipment  has 
less  than  or  equal 
maintenance 
requests.  Stretch 
Goal:  20%  less 

Goal:  Not  met 

6,643%  more  than 
typical 
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3.2  PERFORMANCE  OBJECTIVE  METRICS 


The  origin  of  each  performance  objective  and  its  relevance  to  the  demonstration  are  described 
below.  Each  performance  objective  has  a  metric,  and  all  but  three  have  two  predetermined 
thresholds  for  success:  goal  and  stretch  goal.  The  metric  definitions  and  success  thresholds  for 
each  performance  objective  are  discussed,  and  the  results  for  each  performance  objective  are 
briefly  repeated. 

PO  1:  Improved  Cooling  Energy  Efficiency 

Improved  data  center  cooling  energy  efficiency  is  the  key  advantage  of  the  demonstrated 
technology.  The  energy  needed  to  reject  heat  from  the  IT  equipment  (cooling  energy)  is  typically 
significant,  and  in  some  situations  equal  to  or  more  than  the  energy  used  by  the  IT  equipment. 
The  industry  uses  the  partial  power  usage  effectiveness  (pPUE)  metric  ratified  by  The  Green 
Grid,  and  this  metric  is  used  here. 

Two  quantities  are  required  to  calculate  cooling  pPUE: 

1.  The  energy  consumed  (kWh)  by  equipment  providing  cooling  for  the  data  center 

2.  The  energy  consumed  (kWh)  by  the  IT  equipment 

The  data  used  to  calculate  this  metric  was  from  Romonet  model  energy  simulations.  Romonet  is 
a  UK  software  company  specializing  in  financial  and  energy  modeling  of  data  centers.  See 
Section  6.1  for  more  information. 

The  metric  is  calculated  as  follows: 

pPUE  =  (Quantity  1  +  Quantity  2)  /  Quantity  2  (Eq.  3-1) 

Cooling  technologies  available  in  the  last  5  to  10  years  have  promised  to  reduce  cooling  energy 
use  inside  data  centers  significantly.  The  50  percent  reduction  goal  (of  the  cooling  part — that  is, 
pPUE-1)  was  estimated  to  be  achievable  considering  that  the  demonstrated  technology  does  not 
require  compressor-based  cooling  that  was  assumed  to  be  found  at  a  legacy  DoD  data  center.  The 
success  criteria  (<  50  percent)  was  met  with  a  reduction  of  72  percent. 

PO  2:  Reduced  Overall  Data  Center  Site  Energy  Consumption 

Improved  overall  data  center  energy  efficiency  is  the  key  advantage  of  the  demonstrated 
technology.  While  the  cooling  energy  may  be  reduced  as  described  above,  understanding  the  net 
overall  energy  savings  for  the  whole  data  center  is  the  final  measure  of  savings.  The  industry 
uses  the  PUE  metric  ratified  by  The  Green  Grid,  and  this  metric  is  used  here. 

Two  pieces  of  data  required: 

1.  The  energy  consumed  (kWh)  by  the  whole  data  center 
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2.  The  energy  consumed  (kWh)  by  the  IT  equipment 


The  data  used  to  calculate  PUE  was  generated  by  Romonet  simulation  models.  See  Section  6  for 
more  information. 

The  metric  value  is  calculated  as  follows. 

PUE  =  (Quantity  1  +  Quantity  2)  /  Quantity  2  (Eq.  3-2) 

The  15  percent  reduction  goal  (of  the  infrastructure  part,  that  is,  PUE-1)  was  thought  to  be  easily 
achievable  considering  the  demonstrated  technology  should  provide  a  significant  cooling  energy 
reduction.  The  success  criteria  (>  15  percent)  was  met  with  a  reduction  of  19  percent. 

PO  3:  Improved  Computational  Energy  Efficiency 

This  performance  objective  was  evaluated  to  determine  whether  the  CPU  performance  was 
affected,  while  the  IT  equipment  was  immersion  cooled. 

Two  pieces  of  information  are  required: 

1.  Average  computing  performance  MFLOPS 

2.  Average  power  consumed  (kW)  during  the  performance  test  period 
The  metric  is  calculated  as  follows. 

Compute  Efficiency  =  Quantity  1  /  Quantity  2  (Eq.  3-3) 

If  immersion  cooling  had  better  or  equal  compute  efficiency  compared  to  the  Base  Case,  the  goal 
would  be  met.  The  Pilot  Case  (immersion  cooling)  was  5.6  percent  worse  than  the  Base  Case; 
therefore,  neither  the  goal  nor  the  stretch  goal  were  met. 

PO  4:  Low  Concentration  of  Novec  649  Vapors  During  Normal  Operation 

This  performance  objective  was  evaluated  as  a  safety  precaution.  There  are  periods  when  the  lid 
needs  to  be  opened  exposing  the  operator  to  649  vapors.  The  long-term  exposure  guidelines  for 
Novec  649  for  humans  is  listed  in  the  SDS  (SDS  2015a).  The  goal  was  zero  periods  above 
150  ppmV,  calculated  for  an  8-hour  time- weighted  average  (TWA),  and  the  stretch  goal  was  75 
ppmV. 

Vapor  concentration  measurements  were  recorded  every  five  minutes  and  the  8-hour  TWA 
(indicated  as  “8-Hour  Rolling  Average  PPM”)  values  were  calculated  and  graphed 
(Figure  6.4-2).  The  maximum  8-hour  TWA  concentration  was  48  ppmV,  and  therefore  both  the 
goal  and  the  stretch  goal  for  this  performance  objective  were  met. 
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PO  5:  Low  Concentration  of  Novec  649  Vapors  During  Startup  or  Maintenance 

This  performance  objective  was  also  evaluated  as  a  safety  precaution.  People  may  be  exposed  to 
high  concentrations  during  the  initial  filling  of  the  bath.  The  short-term  exposure  guidelines  for 
Novec  649  for  humans  is  listed  in  the  SDS  (SDS  2015a).  Recording  zero  measurements  above 
100,000  ppmV  was  set  as  the  goal  (there  was  no  stretch  goal). 

Vapor  concentration  measurements  were  recorded  every  five  minutes.  The  maximum  measured 
value  during  the  vapor  level  monitoring  period  at  the  breathing  zone  was  200  ppmV ;  therefore, 
the  goal  for  this  performance  objective  was  met. 

PO  6:  Reduction  in  Direct  Greenhouse  Gas  Emissions 

The  primary  reason  for  investigating  the  demonstrated  two-phase  immersion  cooling  technology 
was  to  establish  the  energy  savings,  and  thereby  estimate  the  potential  for  reducing  greenhouse 
gas  emissions. 

Lower  C02e  emissions  for  the  Immersion  Cooling  Case  compared  to  the  Base  Case  was  the  goal 
for  this  performance  objective.  In  retrospect,  this  goal  should  have  been  set  at  the  same  percent 
reductions  (>  15  percent)  at  that  for  Performance  Objective  2. 

Evaluating  this  metric  is  straightforward: 

1.  Determine  the  overall  data  center  electrical  energy  consumed  for  the  Base  Case  and  the 
immersion-cooled  case. 

2.  Use  the  U.S.  Environmental  Protection  Agency  (EPA)  conversion  calculator  to  find  the 
C02e  emission  rate  per  year  in  metric  tons  for  each  of  the  two  cases  and  calculate  the 
percent  reductions. 

The  goal  was  met,  with  the  calculated  CCEe  reduction  of  19  percent  compared  to  the  Base  Case. 

PO  7:  Dielectric  Liquid  Loss 

The  cost  of  3M  Novec  649  is  considerable  in  the  amounts  needed  to  fill  the  two-phase  immersion 
bath  enclosure.  The  bath  used  in  this  demonstration  required  an  initial  fill  of  595  liters  valued  at 
$44,625  at  $75/liter,  which  was  the  price  at  the  time  of  this  report.  Because  the  open-bath 
enclosure  is  not  sealed  100  percent  of  the  time  to  allow  for  maintenance  procedures  there  was  a 
concern  that  some  Novec  649  may  be  lost  through  the  venting  system  or  escape  into  the  room. 
While  a  high  cost  of  initially  filling  the  bath  is  expected,  potential  additional  costs  from  liquid 
loss  were  investigated  by  evaluating  this  performance  objective. 

To  put  the  cost  of  the  liquid  loss  in  perspective,  the  metric  related  the  cost  of  the  liquid  to  the 
cost  of  IT  equipment  energy  use.  The  metric  was  defined  as  a  ratio:  the  cost  of  the  liquid  lost 
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divided  by  the  cost  of  the  IT  equipment  energy  use.  The  goal  and  stretch  goal  for  this  PO  were 
ratios  of  1  percent  and  0.1  percent,  respectively.  In  retrospect,  these  goals  were  too  optimistic, 
considering  that  the  bath  lid  is  removed  during  IT  equipment  service. 

The  liquid  lost  during  the  demonstration  (86.95  liters)  was  estimated  by  adding  two  quantities: 

1.  Liquid  volume  added  during  the  evaluation  period.  This  quantity  (63.85  liters)  was 
determined  by  the  number  of  bottles  emptied  into  the  bath  after  the  initial  fill. 

2.  Liquid  volume  change  required  to  bring  the  liquid  level  at  the  end  of  the  evaluation 
period  to  that  found  at  the  beginning  of  the  period.  This  quantity  (23.10  liters)  was 
determined  by  comparing  the  liquid  levels  at  the  end  and  beginning  of  the  liquid  loss 
evaluation  period  and  calculating  the  volume  needed  to  make  the  volumes  equal. 

The  energy  used  by  the  IT  equipment  was  obtained  from  the  electrical  meter  that  measured  the 
energy  used  (kWh)  by  the  IT  equipment.  The  NRL  reported  that  the  cost  of  electricity  was 
$0.09  per  kWh.  The  cost  for  energy  used  was  easily  calculated  by  multiplying  the  energy  by  the 
specific  energy  cost. 

The  goal  and  stretch  goal  were  not  met.  The  cost  of  the  lost  liquid  ($6,525)  was  368  percent 
more  than  the  cost  of  the  electrical  energy  used  by  the  IT  equipment. 

PO  8:  System  Economics 

The  system  economics  were  evaluated  to  determine  if  the  technology  was  cost  competitive  with 
other  options.  Evaluations  used  a  simple  payback  method.  In  addition,  the  seven-year  net  present 
value  (NPV)  was  calculated  using  the  National  Institute  of  Standards  and  Technology  (NIST) 
building  life  cycle  cost  (BLCC)  calculator.  Both  evaluation  methods  compared  the  Base  Case 
and  the  Immersion  Cooling  Case  retrofitted  into  an  existing  low-efficiency  data  center.  The 
Immersion  Cooling  Case  simulation  assumed  the  addition  of  a  cooling  tower  to  cool  the 
immersion-cooled  IT  equipment. 

Three  key  yearly  costs,  used  in  both  evaluations  (payback  period  and  NPV),  were  initial  capital, 
energy,  and  liquid  loss  for  the  Immersion  Case.  The  economic  analysis  assumed  the  IT 
equipment  was  consuming  an  average  of  909  kW. 

Initial  Capital 

IT  Equipment 

The  initial  capital  costs  included  the  amount  paid  for  the  IT  equipment  used  in  the 
demonstration. 

Cooling  Systems 

The  cooling  systems  costs  included  the  cooling-related  equipment  that  is  required  as  part 
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of  each  case  (Base  Case  and  Immersion  Case).  The  Base  Case  hybrid  cooling  system 
requires  SGI  Cooling  Racks  and  Cooling  Distribution  Units  (CDUs)  (see  Figure  6.11-1, 
in  Section  6).  The  Immersion  Case  requires  the  bath  enclosure  and  support  systems. 

Novec  649  Initial  Fill 

The  liquid  needed  to  initially  fill  the  bath  was  determined  by  counting  the  number  of 
empty  large  barrels  at  NRL.  The  cost  of  3M  Novec  649  was  considerable.  The  volume 
needed  to  fill  the  two-phase  immersion  bath  enclosure  was  595  liters.  The  total  value  of 
the  liquid  was  $44,625  at  $75/liter,  which  was  the  price  at  the  time  of  this  report. 

Energy 

The  yearly  energy  cost  was  determined  by  using  the  results  from  Performance  Objective  2  and 
an  electrical  energy  cost  of  $0.09/kWh. 

Liquid  Loss 

The  liquid  lost  during  the  demonstration  was  estimated  using  two  quantities: 

1.  Liquid  volume  lost  during  the  loss  evaluation  period 

2.  Liquid  volume  change  required  to  bring  the  liquid  level  found  at  the  end  of  the  loss 
evaluation  period  to  that  found  at  the  beginning  of  the  period 

The  amount  of  liquid  lost  during  the  liquid  loss  evaluation  period  was  obtained  by  counting  the 
number  of  empty  “make-up”  barrels  at  the  end  of  the  period. 

The  amount  of  liquid  needed  to  equalize  the  liquid  level  at  the  end  of  the  period  to  the  beginning 
of  the  period  was  estimated  by  comparing  the  level  readings.  The  volume  of  liquid  was 
calculated  by  subtracting  the  beginning  reading  from  the  end  reading.  More  details  are  provided 
in  Section  6.8. 

The  three  components  discussed  above  were  used  to  calculate  the  simple  payback  period  of 
33  years.  Therefore  neither  the  goal  (<  4  years)  nor  the  stretch  goal  (<  3  years)  were  met. 

PO  9:  Lower  CPU  Chip  Temperatures 

Elevated  CPU  chip  temperatures  have  been  related  to  reduced  reliability  and  higher  energy 
consumption.  Therefore,  a  primary  concern  when  investigating  an  IT  equipment  cooling 
technology  is  the  effect  it  may  have  on  the  operating  temperatures  of  electronic  components, 
specifically  the  CPU.  Exactly  how  higher  temperatures  affect  reliability  and  energy  consumption 
changes  with  each  generation  of  CPU  components.  The  concern  is  founded  on  the  assumed,  but 
not  proven,  phenomena  that  an  increase  in  CPU  temperatures  will  reduce  the  reliability  and 
increase  the  energy  consumption  of  the  CPU  to  an  extent  that  would  affect  business  decisions. 

Using  specific  software,  the  Intel  Xeon  E5-2690  CPU  used  in  this  demonstration  is  able  to  report 
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the  internal  temperature.  During  the  Base  Case  and  Pilot  (also  immersion-cooled)  tests  the 
temperatures  for  the  144  CPUs  were  recorded.  Since  the  CPU  temperatures  were  not  successfully 
measured  for  the  Immersion-Cooled  Case,  the  data  for  the  Pilot  Test  served  as  a  valid  surrogate. 
The  mean  values  for  the  Base  Case  and  Pilot  Tests  were  compared  to  determine  if  the  goal  for 
this  performance  objective  was  met.  The  goal  would  be  met  if  the  immersion-cooled  CPU 
temperature  was  equal  to  or  lower  than  the  Base  Case.  The  mean  value  for  the  immersion-cooled 
equipment  was  20°C  higher  than  for  the  Base  Case,  therefore  the  goal  was  not  met. 

PO  10:  Higher  User  Satisfaction,  Low  Number  of  Concerns 

The  technical  specifications  of  a  data  center  technology  are  very  important,  and  provide 
objective  information  for  potential  buyers.  However,  how  the  user  must  interact  with  the  new 
technology  day  to  day  is  also  an  important  consideration.  For  example,  if  additional  maintenance 
activities  are  associated  with  safety  and  operational  concerns,  the  potential  buyer  may  reconsider 
a  purchase. 

This  performance  objective  was  used  to  find  out  whether  the  user  had  any  safety  or  operational 
concerns  that  were  not  uncovered  while  the  technical  specifications  were  being  originally 
investigated,  and  whether  those  concerns  were  resolved.  The  user  provided  responses  on  four 
subjects: 

•  Safety  concerns 

•  Safety  concerns  that  were  not  resolved 

•  Operational  concerns 

•  Operational  concerns  that  were  not  resolved 

The  goal  was  zero  unresolved  safety  concerns.  The  goal  was  met.  The  stretch  goal  would  be  met 
if  the  goal  was  met  and  there  were  zero  unresolved  operational  concerns.  There  were  13 
operational  concerns  that  were  not  resolved.  Therefore,  the  stretch  goal  was  not  met. 

PO  11:  Improved  IT  Power  Density 

Investigating  the  floor-space  density  is  a  key  consideration  when  a  data  center  cooling 
technology  is  being  considered.  Data  center  operators  need  to  know  the  current  and  future  floor 
space  capabilities  for  a  technology  being  considered.  For  this  performance  objective,  we  define 
density  as  the  number  of  processor  sockets  per  square  foot  of  data  center  floor  space.  Processor 
sockets  are  connectors  on  motherboards  that  hold  the  CPU  components  and  provide  the 
connections  from  the  CPU  component  to  the  motherboard  circuits.  The  compute  capability  of 
an  HPC  computer  cluster  is  almost  entirely  driven  by  the  number  and  model  of  the  connected 
CPU  components. 

This  performance  objective  was  evaluated  by  comparing  the  floor  space  requirements  from  the 
most  dense  SGI  ICE  X  M-Rack  (the  same  model  as  the  Base  Case)  layout  defined  by  SGI  to  a 
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layout  using  the  dimensions  of  the  demonstrated  immersion  bath.  Additional  details  are  provided 
in  Section  6.11. 

The  layout  for  the  Immersion  Case  needed  to  be  more  dense  than  the  currently  offered  density  in 
order  to  meet  the  goal.  However,  the  density  of  the  Immersion  Case  was  only  22  percent  of  that 
of  the  existing  SGI-endorsed  technology  used  for  the  Base  Case.  The  goal  was  not  met. 

There  are  a  number  of  reasons  the  density  for  the  Immersion  Case  is  lower  than  the  layout 
endorsed  by  SGI  using  the  existing  cooling  technology.  One  reason  is  that  the  bath’s  orientation 
is  horizontal  compared  to  the  more  floor  space-efficient  vertical  orientation  of  the  typical  6.5  ft. 
(2  meter)-tall  rack  used  for  the  SGI-endorsed  layout.  Another  reason  is  that  the  demonstrated 
immersion-cooled  configuration  was  cooling  only  one-half  of  an  M-Rack,  compared  to  the  SGI- 
endorsed  layout,  which  uses  a  full  M-Rack  for  each  IT  module. 

PO  12:  System  Maintenance 

Decreased  reliability  and  increased  maintenance,  either  for  the  IT  equipment  or  infrastructure 
systems,  are  primary  concerns  to  research  before  adopting  any  new  data  center  technology. 

The  project  team  compared  the  number  of  maintenance  requests  for  IT  equipment  during  the 
demonstration  to  the  average  number  of  requests  for  the  same  model  IT  equipment  already  sold 
and  in  operation  during  the  past  year. 

The  goal  and  stretch  goal  were:  equal  to  or  better  than  the  typical  maintenance  request  rate  and 
20  percent  lower,  respectively.  The  number  of  maintenance  requests  during  the  demonstration 
were  62  for  7  months  versus  a  typical  1.6  for  the  previous  year.  This  is  an  increase  of 
6,643  percent  ((62/7)*12)/1.6  =  66.43).  The  goal  and  stretch  goal  were  not  met. 

The  large  number  of  IT  equipment  maintenance  requests  were  caused  by  the  large  number  of 
electronic  failures.  More  information  about  these  failures  are  included  in  Section  8.  The  support 
systems  connected  directly  with  the  bath  (such  as  the  liquid  conditioning  pump,  vent  controls, 
and  lid  sealing  pump)  performed  without  failures.  The  only  negative  maintenance  issue  with  the 
bath  was  related  to  the  glass  lid,  which  cracked  twice  and  was  replaced. 
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4.0  FACILITY/SITE  DESCRIPTION 


Three  tests  were  performed  as  part  of  this  ESTCP  project.  First  a  Base  Case  test  was  completed 
to  establish  baseline  data  for  the  thermal  and  computing  performance  characteristics  provided  by 
a  readily  available  cooling  system  using  an  off-the-shelf  HPC  computer  by  SGI.  The  Base  Case 
test  was  conducted  in  a  prototype  assembly  area  inside  a  SGI  manufacturing  facility  located  in 
Chippewa  Falls,  Wisconsin. 

Second,  a  Pilot  Test  was  performed  in  a  different  SGI  building,  also  located  in  Chippewa  Falls. 
This  test  used  the  same  IT  hardware  as  the  Base  Case  test  but  utilized  OBI  immersion  cooling. 
The  Pilot  Test  was  the  first  time  this  electronic  equipment  had  been  cooled  by  the  OBI  cooling. 
The  test  was  performed  at  SGI  in  case  unexpected  startup  or  other  issues  arose  with  the  IT 
equipment.  A  high  computing  load  test  was  also  performed  during  the  Pilot  Test. 

Third,  the  Immersion  Case  testing  took  place  in  a  U.S.  Naval  Research  Faboratory  (NRF)  data 
center  in  Washington,  D.C. 

The  Base  Case  test  was  performed  using  a  single  SGI  ICE  X  “M-Rack,”  with  the  lower  half 
populated  with  two  individual  rack  units  (IRUs)  (Figure  4-1).  A  cooling  distribution  unit  supplies 
water  that  directly  cools  the  144  processors  located  within  the  two  IRUs.  The  remaining  heat 
(10-30  percent)  is  removed  by  air  cooling  provided  by  a  cooling  rack  (Figure  4-1)  that  is  placed 
between  two  M-Racks.  This  hybrid  cooling  system,  consisting  of  direct  water  and  air  cooling,  is 
designed  to  be  room  neutral ,  which  means  that  nearly  all  the  heat  from  the  IT  equipment  is 
captured  locally  (at  the  rack  level)  and  very  little  heat  needs  to  be  cooled  by  the  room  air 
conditioning  system.  Therefore,  capturing  detailed  room  air  conditioning  data  was  not 
considered  an  important  factor  for  performing  a  valid  Base  Case  test. 
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Figure  4-1:  The  Base  Case  Test  Setup 

The  Pilot  Test  was  set  up  in  a  basement  room  (Figure  4-2)  with  limited  air  conditioning 
capability.  This  test  was  also  assumed  to  be  room  neutral  because  it  was  assumed  that  a  high 
percentage  of  the  IT  equipment  heat  load  would  be  removed  by  the  OBI  cooling.  Temperature 
variation  in  the  room  was  not  noticeable,  indicating  that  the  existing  limited  air  conditioning 
was  sufficient. 
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Figure  4-2:  Pilot  Test  Setup  at  SGI 

The  demonstration  test  was  conducted  in  a  data  center  room  at  the  NRL  in  Washington,  D.C. 


4.1  FACILITY/SITE  LOCATION  AND  OPERATIONS 

For  the  demonstration  test,  the  OBI  bath  was  located  inside  a  data  center  room  at  NRL.  A  dry- 
cooler  type  of  heat  exchanger  located  immediately  outside  the  building  provided  the  heat 
rejection  needed  for  the  immersion  bath  (Figure  4-3). 

This  data  center  facility  is  operated  around  the  clock:  24  hours  per  day,  7  days  per  week.  Except 
for  this  demonstration,  the  data  center  was  operated  normally.  On  a  typical  work  day,  a  small 
number  of  people  occupy  the  data  center  to  monitor,  configure,  or  service  the  IT  equipment. 
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Figure  4-3:  Dry  Cooler  Outside  of  NRL  Data  Center 

Data  center  room  is  approximately  3,000  square  feet  and  houses  electronic  equipment  that  uses 
approximately  1,750,000  kWh/year.  This  existing  air-cooled  equipment,  which  was  not  used  for 
the  demonstration  test,  is  cooled  by  a  traditional  underfloor  air  plenum  with  computer  room  air 
conditioners  (CRACs).  The  temperature  in  the  center,  at  the  air  inlets  to  the  electronic 
equipment,  is  controlled  and  typically  ranges  from  69°F  (21°C)  to  73°F  (23°C). 

Figure  4-4  shows  the  bath  installed  at  NRL  with  the  FT  equipment  powered  on.  A  data  center 
designed  exclusively  for  immersion  cooling  would  probably  have  airflow  velocities  near  the  bath 
much  lower  than  is  typical  with  a  raised-floor  cooled  data  center.  Therefore,  a  plastic  curtain 
containment  system  was  installed  (visible  in  Figure  4-5)  to  reduce  air  currents  that  might 
increase  Novec  649  vapor  loss  when  the  tank  lid  was  open  during  service  or  startup  operations. 
The  cooling  water  supply  and  return  for  the  condenser  coils  inside  the  bath  were  provided  from 
overhead  via  two  2-inch-diameter  copper  pipes  (Figure  4-5). 
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Figure  4-4:  Bath  Located  in  the  NRL  Data  Center  Room 
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4.2  FACILITY/SITE  CONDITIONS 


The  site  was  selected  primarily  by  NRL’s  desire  to  be  part  of  this  demonstration.  The 
Washington,  D.C.,  climate  can  provide  conditions  that  challenge  the  energy  efficiency  of 
conventional  cooling  systems.  For  example,  there  are  many  days  during  the  summer  when  the 
dry-bulb  temperature  is  high,  with  coincident  high  relative  humidity.  Data  center  sites  that  have  a 
cooling  tower  and  chiller  cooling  infrastructure  need  to  be  sized  to  provide  enough  cooling 
during  these  worst-case  conditions.  Therefore,  the  Washington,  D.C.,  climate  is  a  good  candidate 
for  exploring  the  energy  savings  that  might  be  provided  by  a  technology  that  reports  to  offer 
significant  cooling  system  energy  savings. 
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5.0  TEST  DESIGN 


This  section  describes  the  demonstration  test  design  used  to  obtain  data  to  support  the  evaluation 
of  the  performance  objectives.  The  general  approach  was  to  compare  the  energy  efficiency 
performance  of  the  immersion  cooling  technology  to  commercial  technology  available  from  SGI. 
In  addition,  a  number  of  other  parameters  not  directly  connected  to  energy  efficiency  were 
monitored  in  the  event  that  those  parameters  were  affected  by  the  immersion  cooling  technology. 

5.1  CONCEPTUAL  TEST  DESIGN 

The  measurements  for  evaluating  the  performance  objectives  were  gathered  during  three  distinct 
tests.  The  Base  Case  and  the  Pilot  Test  were  performed  at  SGI  in  Wisconsin.  The  Immersion 
Case  test  was  located  in  a  data  center  at  the  Naval  Research  Laboratory  (NRL)  in  Washington, 
D.C.  All  three  test  phases  used  the  same  IT  equipment  from  SGI.  The  Base  Case  used 
commercial  cooling  technology  from  SGI,  whereas  the  Pilot  Test  and  the  Immersion  Case  used 
the  demonstrated  immersion  cooling  (OBI). 

Base  Case  Test 

Base  Case  testing  was  performed  inside  a  SGI  manufacturing  building  in  Chippewa  Falls, 
Wisconsin.  This  test  was  used  to  gather  thermal  and  computational  performance  data  using  the 
water-cooled  technology  commercially  available  from  SGI.  The  IT  equipment  tested  was  half  of 
an  SGI  ICE  X  M-Rack  (Figure  5-1).  This  half  rack  consisted  of  36  SGI  ICE  X  blades  containing 
two  node  boards  each.  Each  node  board  contained  two  CPUs  and  four  memory  dual  in-line 
memory  modules  (DIMMs)  per  CPU.  The  total  number  of  key  components  therefore  consisted  of 
144  CPUs  and  576  memory  DIMMs. 
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IT  Equipment  SGI 

Under  Test  Cooling 

(>2  M  Rack)  Rack 


Figure  5-1:  Base  Case  Test  Configuration 

The  Base  Case  test  consisted  of  gathering  data  during  a  period  of  approximately  one  hour  during 
constant  thermal  and  controlled  computational  load  conditions.  The  test  was  controlled  by 
providing  a  constant  IT  equipment  load  during  a  period  where  the  supplied  building-cooling 
water  temperature  was  nearly  constant.  The  computational  performance  results  were  recorded 
using  common  high-performance  computing  (HPC)  benchmark  software  called  UNPACK.  The 
electrical  power  needed  for  the  IT  equipment  and  associated  cooling  systems  were  periodically 
recorded.  The  test  results  were  used  to  establish  the  performance  objective  baselines. 

Pilot  Test 

The  Pilot  Test  was  performed  in  the  basement  (Figure  5-2)  of  a  SGI  engineering  building  also 
located  in  Chippewa  Falls,  Wisconsin.  The  test  was  performed  at  SGI  in  the  event  start-up  issues 
required  SGI  technical  expertise.  The  Pilot  Test  used  the  same  half  rack  of  SGI  ICE  X  super¬ 
computer  hardware  employed  in  the  Base  Case  test  but  was  immersed  and  cooled  using  3M 
Novec  649  in  a  bath  enclosure  constructed  by  3M.  This  bath  would  later  be  moved  to  NRL  to 
perform  the  Immersion  Case  test. 
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Figure  5-2:  Pilot  Test  -  IT  Running  at  Full  Power. 

Note:  Boiling  Novec  649  appears  white.  Novec  649  liquid  and  vapor  are  transparent. 

Immersion  Case  Test 

In  September  2014,  the  bath  and  IT  equipment  was  moved  from  the  Pilot  Test  location  to  the 
Immersion  Case  test  location  and  installed  (Figure  5-3)  in  the  data  center  space  at  the  NRL  in 
Washington,  D.C.  The  tests  were  performed  from  October  2014  to  August  2015.  The  cooling 
system  was  designed  to  provide  different  supply-water  temperatures  (Figure  5-4,  "Temperature 
Control"),  in  order  to  obtain  data  needed  to  simulate  long-term  performance  in  different  climates. 
The  cooling  system  also  had  a  selectable  pressure  difference  across  the  bath  condenser  coil,  as 
might  be  encountered  in  a  data  center  containing  multiple  immersion  cooling  baths.  In  such  a 
configuration  the  flow  rate  could  be  adjusted  for  each  bath  using  a  two-way  valve. 
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Figure  5-3:  Immersion  Case  Test  Installation  (Demonstration) 


Figure  5-4:  Immersion  System  Cooling  with  a  Dry  Cooler 
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5.2  BASELINE  CHARACTERIZATION 


The  baseline  characterization  was  performed  with  the  Base  Case,  which  used  a  commercial 
hybrid  (liquid-air)  cooling  system  supplied  by  SGI.  The  cooling  system  used  direct  liquid 
cooling  for  the  processor  components.  Cooling  distribution  units  provided  the  required  cooling 
water.  Forced-air  cooling  was  used  for  the  remaining  electronic  and  electrical  components.  The 
forced-air  cooling  was  provided  by  an  SGI  brand  Cooling  Rack  (Figure  5-1). 

The  CDU  and  Cooling  Rack  transfer  virtually  all  the  heat  from  the  IT  equipment  to  the  building 
water  cooling  system.  The  water  temperatures  supplied  during  the  Base  Case  test  were  used  in 
simulations  that  compared  the  Base  Case  and  the  Immersion  Case.  The  Base  Case  test  took  place 
during  a  one-hour  period  while  the  FT  equipment  was  running  at  a  very  high  load  provided  by  the 
UNPACK  software.  The  one-hour  period  provided  data  for  a  single  set  of  operational 
parameters.  Data  for  other  cooling  water  temperatures  or  different  IT  equipment  loads  were  not 
collected. 

Data  collection  points  for  the  Base  Case  test  (see  Figure  5-5): 

•  SGI  cooling  system  pump  (P3)  and  fan  power  (PI) 

•  IT  equipment  power  (P2) 

•  Cooling  Water  Supply  Temperatures  (T3,  T6) 

•  Cooling  Water  Return  Temperature  (T7) 

•  Cooling  Water  Supply  Flow  Rates  (FI,  F2) 

•  Computational  Performance  (reported  separately) 

The  energy  consumption  and  thermal  parameter  measurements  and  calculated  results  from  the 
Base  Case  test  are  listed  in  Tables  D-l  through  D-4. 
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Figure  5-5:  Base  Case  Test  Sampling  Point  Schematic 
5.3  DESIGN  AND  LAYOUT  OF  TECHNOLOGY  COMPONENTS 

The  layout  of  the  technology  components  will  be  presented  in  two  sections:  (1)  the  immersion 
tank  interface  with  cooling  water  supply,  and  (2)  details  on  the  systems  and  components  directly 
attached  to  the  immersion  bath. 

Immersion  Tank  Cooling  Interface 

The  supply  of  cooling  water  can  be  provided  from  any  clean  source,  as  long  as  that  source  can 
supply  the  water  temperature  and  flow  rate  required  to  meet  the  IT  equipment  heat  load.  A  “dry 
cooler”  is  an  efficient  source  for  cooling  the  water  and  was  used  in  the  Immersion  Case 
simulations  along  with  two  other  heat-rejection  technologies  (building  water  and  cooling  tower). 
A  dry  cooler  can  be  implemented  using  a  simple  piping  and  control  scheme  (Figure  5-4). 

Immersion  Tank  Systems  and  Components 

The  immersion  bath  consisted  of  a  number  of  components  and  subsystems  (Figure  5-6),  which 
are  described  in  the  text  following  the  figure. 
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Figure  5-6:  Bath  Components  and  Subsystem  Reference  Diagram 


1.  IT  Equipment:  The  IT  equipment  (#1)  is  completely  submerged  in  the  3M  Novec  649 
liquid  at  all  times.  If  the  liquid  level  falls  to  a  point  where  the  IT  equipment  is  exposed  to 
the  headspace,  the  power  to  the  equipment  is  automatically  turned  off.  The  IT  equipment 
is  located  directly  below  the  removable  lid  (#4),  which  is  sized  such  that  equipment  can 
be  installed  and  removed  for  service. 

2.  Primary  Condenser:  The  primary  condenser  (#2)  is  constructed  with  brazed  copper 
tubes.  Water  flowing  through  the  primary  condenser  removes  heat  from  the  Novec  649 
vapor  by  condensing  the  vapor  back  to  a  liquid.  The  liquid  falls  back  by  gravity  into  the 
bath,  and  no  pumping  is  required.  An  external  valve  (not  shown)  is  used  to  modulate  the 
water  flow  rate  through  the  primary  condenser  to  keep  the  vapor  height  within  desired 
limits. 

3.  Hot  Swap  System:  During  operation,  the  temperature  of  the  water  entering  the  primary 
condenser  is  typically  warmer  than  the  ambient  air  outside  the  bath.  This  means  that 
Novec  649  vapor  can  condense  on  the  lid  thereby  reducing  visibility.  When  removing  the 
lid  to  service  IT  components,  the  lid  may  be  wet  with  Novec  649  droplets,  and  much  of 
that  liquid  could  be  lost  to  evaporation. 

The  Hot  Swap  System  helps  resolve  the  condensation  issue.  The  Hot  Swap  System 
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includes  a  small  pump,  radiator,  and  fan  (these  items  not  shown)  connected  to  a 
condenser  (#3)  inside  the  tank  above  the  primary  condenser.  Before  the  lid  is  opened, 
water  at  room  temperature  is  initiated  through  the  secondary  coil,  causing  the  Novec  649 
vapor  to  condense  and  fall  back  into  the  bath. 

4.  Removable  Lid:  A  removable  lid  (#4)  with  an  inflatable  seal  provides  an  opening 
needed  to  service  any  component  located  in  the  bath.  The  sealing  performance  of  the  lid 
is  important,  as  significant  amounts  of  Novec  649  vapor  mixed  with  air  in  the  headspace 
can  escape  without  easily  being  detected. 

5.  Level  Sensor:  The  liquid  level  is  reduced  if  Novec  649  evaporates.  This  can  occur 
when  the  lid  is  removed  to  add  or  remove  equipment.  A  level  sensor  (#5)  is  needed  to 
warn  the  operator  when  the  liquid  level  is  too  low  or  too  high.  An  emergency  power  off 
action  was  programmed  using  the  level  sensor  readings  to  automatically  turn  off  the  IT 
equipment  power  should  the  liquid  level  be  too  low  for  adequate  cooling. 

6.  Organic/Particulate  Filtration:  Components  in  electronic  equipment,  such  as  cabling, 
contain  small  levels  of  organic  oils.  These  oils  are  easily  extracted  by  the  Novec  649 
liquid  and  can  subsequently  be  deposited  by  distillation  on  or  near  the  boiling  surfaces. 
This  process  may  degrade  the  heat  transfer  performance.  Activated  carbon  removes  such 
contaminants.  A  pump  draws  liquid  through  a  commercially  available  carbon  cartridge 
(#6),  which  also  removes  particles  that  might  be  suspended  in  the  liquid. 

7.  Moisture  Control  System:  At  startup,  the  Novec  649  liquid,  as  well  as  the  electronics, 
are  likely  to  contain  a  small  amount  of  water.  The  boiling  process  vaporizes  this  water, 
which  collects  in  the  headspace.  A  fan  recirculates  the  gases  in  the  headspace  through  a 
renewable  desiccant  (#7)  to  control  this  moisture.  The  amount  of  desiccant  is  sized  to 
minimize  the  replacement  frequency.  The  desiccant  changes  colors  when  it  is  in  need  of 
regeneration. 

8.  9,  10,  and  12.  Vent  and  Pressure  Control  Subsystem:  This  subsystem  maintains 
atmospheric  pressure  in  the  headspace  while  minimizing  vapor  losses.  If  the  power 
consumed  by  the  IT  equipment  (#1)  increases  and/or  the  flow  of  cooling  water  decreases, 
the  vapor  level  can  rise  considerably.  As  the  vapor  level  rises,  the  air  and  possibly  some 
vapor  in  the  headspace  may  need  to  be  vented  outside  of  the  facility  via  the  vent-out 
valve  (#10).  The  vent  and  pressure  control  subsystem  also  includes  a  bellows  (#8)  to 
accommodate  small  changes  in  the  vapor  height  during  normal  operation.  A  vapor 
recovery  system  (#9)  condenses  vapor  that  may  escape  and  returns  it  to  the  bath. 
Mechanical  pressure  and  vacuum  check  valves  (not  shown)  will  act  as  backup  to  provide 
needed  venting  in  the  case  of  a  vent  component  or  power  failure.  Similarly,  the  bath  will 
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take  in  air  from  the  outside  if  a  significant  vacuum  develops  by  opening  the  vent-in 
valve  (#12). 

11.  Secondary  Containment:  To  contain  a  leak  or  spill  of  liquid  Novec  649,  a  secondary 
containment  feature  (#11)  is  necessary.  The  volume  of  the  secondary  containment  system 
was  capable  of  handling  a  worst-case  spill  event. 

5.4  OPERATIONAL  TESTING 

The  operational  testing  of  immersion  cooling  was  separated  into  two  phases: 

1.  Pilot  Case  testing  (at  SGI) 

2.  Immersion  Case  testing  (at  NRL) 

Pilot  Case  Testing  (at  SGI) 

The  Pilot  Case  testing  period  was  from  August  27,  2014,  to  September  12,  2014.  The 
measurements  for  evaluating  Performance  Objective  3  (Computational  Energy  Efficiency)  and  9 
(CPU  Chip  Temperatures)  were  obtained  during  this  test. 

The  data  collection  part  of  the  test  was  run  for  approximately  one  hour  after  steady-state 
conditions  were  achieved.  The  computational  performance  was  reported  by  the  UNPACK 
software  (Table  6.3-1).  In  addition,  during  the  test,  CPU  temperatures  (Figure  6.9-2)  and  energy 
consumption  recorded. 

Immersion  Case  Testing  (at  NRL) 

The  testing  at  NRL  was  split  into  two  parts:  Startup  and  normal  operation.  The  ten-day  startup 
period  (October  7,  2014,  to  October  17,  2014)  included  filling  the  tank  with  3M  Novec  649,  as 
well  as  installing  and  starting  the  IT  equipment  and  the  data  collection  system.  Normal  operation 
started  on  October  17,  2014,  and  ended  in  August  5,  2015. 

Unexpected  failures  of  immersion-cooled  IT  equipment  limited  the  planned  testing,  particularly 
the  thermal-  and  energy-related  tests.  The  measurements  were  to  be  used  for  calculating  the 
metrics  of  Performance  Objectives  1,  2,  6,  and  8.  Although  modeling  was  originally  planned  as 
an  integral  part  of  estimating  the  metrics,  the  lack  of  data  due  to  these  equipment  failures 
increased  reliance  on  modeling  even  more. 

Time-series  measurements  were  obtained  over  the  10-month  test  period  to  support  calculations 
for  liquid  loss  and  vapor  concentration  monitoring.  In  addition,  the  cooling  energy  effectiveness 
(pPUE)  was  calculated  for  a  limited  period  (Figure  6.1-3)  to  highlight  the  technology’s  energy 
efficiency  potential.  However,  these  results  were  not  used  to  evaluate  the  performance 
objective  metrics. 
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Liquid  Loss 

A  liquid  level  sensor  was  used  to  record  the  Novec  649  liquid  level  in  the  bath.  The 
measurements  were  combined  with  the  record  of  liquid  added  during  the  liquid-loss 
evaluation  period  to  determine  the  net  liquid  loss.  The  liquid  level  measurements  are 
graphed  in  Figure  6.7-1.  In  Figure  6.7-1  a  range  of  100  on  the  Tank  Liquid  Level  axis 
corresponds  to  11.52  liters. 

Novec  649  Vapor  Concentrations 

The  Novec  649  vapor  concentrations  were  measured  at  two  locations:  (1)  where  a 
technician  would  be  during  bath  maintenance  procedures,  and  (2)  under  the  raised  floor 
adjacent  to  the  bath.  The  measurements  from  the  under- floor  location  (Figure  6.4-1 
bottom  graph)  were  always  lower  than  those  taken  from  the  breathing  zone  (Figure  6.4-1 
top  graph).  Therefore  the  data  from  the  breathing  zone  were  analyzed  to  calculate  the 
8-hour  TWA  values  (Figure  6.4-2  top  graph)  used  to  determine  the  final  metric  value. 

Figure  6.5-1  shows  the  vapor  concentration  measurements  at  the  breathing  zone  for  the 
startup  period. 

Cooling  Energy  Efficiency 

The  measurements  needed  to  calculate  the  actual  immersion  cooling  energy  efficiency 
were  recorded  for  a  limited  time  only.  The  measured  cooling  energy  constituent  are 
graphed  in  Figure  6.1-3  as  “Dry  Cooler  Fan  Power,”  “Water  Loop  Pumping  Power,”  and 
“Tank  Systems  Power.” 


5.5  SAMPLING  PROTOCOLS 

Figure  5-5  presented  the  Base  Case  thermal,  flow,  and  power  sampling  points.  Table  5-1  lists  the 
sampling  point  details.  The  measurements  were  obtained  manually  except  for  the  compute 
performance  results,  which  were  provided  as  a  report  file  by  the  UNPACK  software. 


Table  5-1:  Base  Case  Test  Sampling  Point  Details 


Description 

Source 

Units 

Result 

IT  Equipment  Power  [P2] 

visual  reading 

kW 

28.4 

Cooling  Rack  Supply  [T3]  Temperature 

visual  reading 

°C 

19.88 

Cooling  Rack  Electrical  Power  [PI] 

visual  reading 

kW 

2.04 

Cooling  Rack  Flow  Rate  [F2] 

visual  reading 

gpm 

24.31 

Direct  Cooling  Supply  Temperature  [Tl] 

visual  reading 

°F 

67.32 

Direct  Cooling  Flow  Rate  [FI] 

visual  reading 

gpm 

68.86 
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Direct  Cooling  Pump  Power  [P3] 

visual  reading 

kW 

3.23 

CDU  Supply  Temperature  [T6] 

visual  reading 

°C 

19.35 

Room  Dry  Bulb  Temperature 

data  from  cooling 
rack 

°C 

19.66 

Room  Dew  Point  Temperature 

data  from  CDU 

°c 

7.1 

gpm  =  gallons  per  minute 


The  Immersion  Case  thermal,  flow,  and  power  sampling  points  data  (Figure  5-7)  were  recorded 
in  a  database  (see  Table  5-2).  Table  5-2  lists  the  sampling  point  details  used  to  calculate  an  actual 
cooling  pPUE  for  a  limited  period. 


database 


local  dashboard 

remote  dashboard 
remote  data/  reports 


Figure  5-7:  Immersion  Case  Test  Sampling  Point  Schematic 
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Table  5-2:  Immersion  Case  Test  Sampling  Point  Details 


Description 

Source 

Units 

Results 

[FTP] 

power  to  IT  equipment 
in  the  bath 

PowerScout  meter 
Modbus 

kW,  kWh 

See  measurements  for 
Performance  Objective 

1,  Figure  6.1-3 

[LEVEL] 
liquid  level 

Level  Sensor 

3M  Controls 

sensor  value 

See  measurements  for 
Performance  Objective 

7,  Figure  6.7-1 

[PUMPP] 

cooling  pump  power 

PowerScout  meter 
Modbus 

kW 

See  measurements  for 
Performance  Objective 

1,  Figure  6.1-3 

[DCP] 

dry  cooler  fan  power 

PowerScout  meter 
Modbus 

kW 

See  measurements  for 
Performance  Objective 

1,  Figure  6.1-3 

[CONP] 

bath  controls  power 

PowerScout  meter 
Modbus 

kW 

See  measurements  for 
Performance  Objective 

1,  Figure  6.1-3 

[OUTTH] 
temperature  and 
humidity  near  dry 
cooler 

sensor 

Modbus 

Temperature  (°C) 

See  measurements  for 
Performance  Objective 

1,  Figure  6.1-3 

Equipment  Calibration 

All  the  meters  used  for  measuring  vapor  concentration,  electrical  power,  water  flow  rate,  and 
water  temperature  were  purchased  new  for  this  project. 

The  Novec  649  liquid-level  sensor  was  calibrated  with  actual  measurements  taken  in  the  bath. 
See  details  for  this  work  in  Appendix  B. 


5.6  SAMPLING  RESULTS 

Due  to  the  number  of  performance  objectives  the  sampling  results  in  the  form  of  graphs  and 
tables  are  shown  in  Section  6  and  in  Appendices.  For  convenience  Table  5.6-1  is  provided  as  a 
reference. 
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Table  5.6-1:  Report  Locations  of  Data  and  Results  Figures  and  Tables 


Data  Description 

Location 

Cooling  Energy  Simulation  Results 

Table  6.1-2 

Overall  Energy  Simulation  Results 

Table  6.1-2 

Simulation  Component  Result  Details 

Table  C-2 

Computational  Efficiency  Data 

Table  6.3-1 

Novec  649  Normal  Operation  Vapor 

Concentration  Measurements 

Figures  6.4-1.  6.4-2 

Novec  649  Maintenance  Period  Vapor  Concentration 
Measurements 

Figure  6.5-1 

Greenhouse  Gas  Emissions  Results 

Tables  6.6-1,  6.6-2 

Liquid  Level 

Figure  6.7-1 

CPU  Temperature  Data 

Figures  6.9-1,  6.9-2 

Cooling  Energy  from  actual  measurements 

Figure  6.1-3 

Base  Case  Test  Data 

Tables  D-l.D-2,  D-3,  D-4 
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6.0  PERFORMANCE  ASSESSMENT 


This  section  provides  an  overview,  goals,  and  results  for  each  performance  objective. 

6.1  IMPROVED  COOLING  ENERGY  EFFICIENCY  [POl] 

Metric:  Cooling  system  energy  use  based  on  pPUE 

Goal:  50  percent  less  cooling  energy  compared  to  the  Base  Case 

The  metric  evaluations  for  Performance  Objectives  1,  2,  6,  and  8  were  based  on  data  center 
energy  simulations.  The  Romonet  software  was  utilized  to  create  eight  (8)  data  center  models. 

All  models  used  average  weather  data  for  Washington,  D.C.  and  assumed  that  the  average  IT 
power  was  80  percent  of  a  design  maximum  of  2  MW.  The  research  advisors  for  this  project 
suggested  the  80  percent  value  as  a  maximum  average  power  level.  The  2  MW  data  center  was 
selected  because  the  cooling  infrastructure  equipment  would  typically  be  large  enough  to  have 
good  efficiency. 

The  models  assume  a  retrofit  situation  where  80  percent  of  the  IT  equipment  is  being  upgraded 
and  there  are  two  technology  choices  (Base  Case  and  Immersion  Case)  for  the  upgraded 
equipment. 

An  IT  equipment  cooling  technology  will  likely  produce  different  overall  energy  efficiencies, 
depending  on  the  data  center  cooling  infrastructure  of  the  data  center  being  retrofitted.  Two  data 
centers  were  used  in  the  models:  one  high  efficiency  and  one  low  efficiency.  Additional 
information  on  the  two  models  is  provided  in  Appendix  C.  This  created  four  efficiency  model 
combinations  (Figure  6.1-1).  Four  additional  combinations  (Immersion  B2  and  B3,  and 
Immersion  A2  and  A3)  (Figure  6.1-2)  combined  to  make  eight  (8)  total  models  (Table  6.1-1). 
Four  additional  combinations  (Immersion  B2  and  B3,  and  Immersion  A2  and  A3)  (Figure  6.1-2) 
combined  to  make  eight  (8)  total  models  (Table  6.1-1). 

It  was  assumed  that  the  Base  Case  (SGI  hybrid  cooling)  would  be  retrofitted  into  the  data  center 
infrastructure  so  that  the  cooling  water  would  be  provided  by  the  existing  cooling  water  supply 
(heat  rejection).  Appendix  D  contains  details  for  determining  model  input. 

Since  immersion  cooling  can  use  warmer  water  compared  to  the  Base  Case,  three  different  heat 
rejection  methods  were  modeled  (Figure  6.1-2).  Eight  (8)  model  diagrams  from  the  Romonet 
software  are  provided  in  Figures  C-l  through  C-8.  The  corresponding  simulated  combinations 
are  shown  in  Table  6.1-1.  Two  of  the  model  combinations  (Base  Case  A  and  Immersion  Case 
A3),  indicated  with  bold  italics,  were  used  to  evaluate  Performance  Objectives  1,  2,  6,  and  8. 
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The  simulated  results  for  all  combinations  are  provided  in  Table  6.1-2. 


IT  Cooling 

Technology 

Data  Center 
Efficiency 

f  \ 

Base  Case 
(hybrid  cooling) 

r  \ 

High 

L  J 

>< 

r  \ 

Immersion  Case 

(immersion  cooling) 

v  J 

r  \ 

Low 

'v.  J 

Figure  6.1-1:  IT  Cooling  Technology  and  Data  Center  Efficiency  Combinations 


Data  Center 
Heat  Rejection 
Method 


Figure  6.1-2:  Immersion  Cooling  -  Data  Center  Heat  Rejection  Combinations 


Table  6.1-1:  Combinations  Simulated 


Simulation  ID 

Data  Center 
Efficiency  Level 

Heat  Rejection 

Method 

Base  Case  B 

High 

Existing 

Immersion  B 1 

High 

Existing 

Immersion  B2 

High 

Dry  Cooler 

Immersion  B3 

High 

Cooling  Tower 
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Base  Case  A 

Low 

Existing 

Immersion  A1 

Low 

Existing 

Immersion  A2 

Low 

Dry  Cooler 

Immersion  A3 

Low 

Cooling  Tower 

The  Romonet  simulation  results  (Table  C-2)  contain  a  listing  of  average  power  consumed  for 
each  component  shown  in  Figures  C-l  through  C-8.  A  number  of  these  components  are 
providing  cooling  for  the  data  center.  Those  components  are  indicated  on  the  left  as  “Inside 
Cooling’  (components  located  inside  the  building)  and  “Outside  Cooling’’  (components  located 
outside  the  building). 

Equations  6-1  and  6-2  define  the  cooling  partial  PUE  (pPUE): 

Sum  of  cooling  components  =  Inside  Cooling  components  + 

Outside  Cooling  components  (Eq.  6-1) 

pPUE  (cooling)  =  ((Sum  of  cooling  components)  +  IT  power  )  A  IT  power  (Eq.  6-2) 

The  calculations  show  a  73  percent  (Table  6.2-1)  cooling  energy  savings  when  the  efficient 
immersion  cooling  option  (Immersion  Case  A3)  is  compared  to  Base  Case  A,  thereby  meeting 
the  goal  (>  50  percent  savings).  It  is  interesting  to  note  that  the  goal  is  also  met  when  the  choice 
is  applied  to  a  high-efficiency  data  center  (Base  Case  B  versus  Immersion  3B). 

Different  assumptions  will  provide  different  energy  savings  estimates.  For  example,  different 
climates,  data  center  efficiency  level,  heat  rejection  method,  and  percentage  of  IT  equipment 
being  upgraded  will  affect  the  estimates.  Those  wishing  to  follow  the  energy  estimate  methods 
outlined  here  are  strongly  encouraged  to  consult  with  data  center  simulation  experts  and  use 
inputs  appropriate  for  their  particular  situation. 

To  highlight  the  low  cooling  energy  potential  of  immersion  cooling  measurements  were  recorded 
for  a  limited  time  during  the  demonstration  and  were  used  to  calculate  periods  where  the  cooling 
pPUE  of  just  under  1.02  (Figure  6.1-3). 
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Figure  6.1-3:  Calculated  Cooling  pPUE  And  Measured  Constituents 


The  cooling  pPUE  for  the  high  efficiency  data  center  simulation  was  1.06  and  included  the 
cooling  energy  needed  for  the  20  percent  of  air-cooled  IT  equipment.  Therefore,  the  cooling 
pPUE  did  not  approach  the  1.02  seen  during  the  short  test. 


Table  6.1-2:  Simulation  Results  Summary 


Low  Efficiency  Data  Center 

Case  Description 

IT  Power  (kW) 

Infractructure  Power  (kW) 

Air 

Cooled 

Direct 

Cooled 

Immersion 

Cooled 

Inside 

Cooling 

Outside 

Cooling 

Other 

Infrastructure 

Cooling 

pPUE 

PUE 

Cooling 

Energy 

Reduction 

(%) 

Basecase  A 

Low  Efficiency  UPS  and  Chiller 

321 

1275 

66.9 

558 

158 

1.392 

1.491 

Immersion  1A 

Building  Chilled  Water 

321 

1275 

20.3 

543 

158 

1.353 

1.452 

10% 

Immersion  2A 

Add  Dry  Cooler  with  Adiabatic  Feature 

321 

1275 

20.0 

162 

154 

1.114 

1.210 

71% 

Immersion  3A 

Add  Cooling  Tower 

321 

1275 

20.0 

151 

154 

1.107 

1.203 

73% 

High  Efficiency  Data  Center 

Case  Description 

IT  Power  (kW) 

Infractructure  Power  (kW) 

Air 

Cooled 

Direct 

Cooled 

Immersion 

Cooled 

Inside 

Cooling 

Outside 

Cooling 

Other 

Infrastructure 

Cooling 

pPUE 

PUE 

Cooling 

Energy 

Reduction 

(%) 

Basecase  B 

High  Efficiency  UPS  and  Chiller 

321 

1275 

66.4 

138 

140 

1.128 

1.216 

Immersion  IB 

Building  Chilled  Water 

321 

1275 

19.8 

139 

140 

1.099 

1.187 

22% 

Immersion  2B 

Add  Dry  Cooler  with  Adiabatic  Feature 

321 

1275 

19.8 

91 

139 

1.070 

1.157 

46% 

Immersion  3B 

Add  Cooling  Tower 

321 

1275 

19.8 

75 

139 

1.059 

1.146 

54% 
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6.2  REDUCED  OVERALL  DATA  CENTER  SITE  ENERGY  CONSUMPTION 
[P02] 


Metric:  Overall  data  center  energy  (PUE) 

Goal:  15  percent  improvement  compared  to  the  Base  Case 

Stretch  Goal:  20  percent  improvement  compared  to  the  Base  Case 

The  assumptions  and  details  for  the  modeling  and  simulations  are  provided  in  Section  6.1  above 
and  in  Appendix  C. 

The  estimated  percent  of  cooling  energy  savings  as  measured  by  the  pPUE  (Performance 
Objective  1)  does  not  reflect  the  overall  data  center  savings  because  data  centers  contain  systems 
other  than  cooling  that  consume  energy.  To  get  a  perspective  on  the  overall  energy  savings 
associated  with  the  immersion  cooling  technology,  the  often-used  industry  metric  power  usage 
effectiveness  (PUE)  was  used. 

Power  usage  effectiveness  is  defined  as  the  total  data  center  site  energy  consumption  (including 
the  IT  equipment)  divided  by  the  IT  equipment  energy  consumption. 

Table  6.1-1  in  Section  6.1  contains  the  calculated  PUE  values  from  the  eight  (8)  Romonet 
simulations. 

The  same  two  cases  (Base  Case  A  and  Immersion  Case  3A)  were  used  for  evaluating  this 
performance  objective.  The  PUE  results  were  1.491  for  Base  Case  A  and  1.203  for  Immersion 
Case  3 A.  These  data  result  in  a  19  percent  savings  on  the  total  data  center  energy. 

It  is  interesting  to  note  that  the  PUE  results  using  a  high-efficiency  data  center  were  1.216  for 
Base  Case  B  and  1.146  for  Immersion  Case  3B.  These  data  result  in  a  6  percent  savings  on  the 
total  data  center  energy.  This  demonstrates  the  law  of  diminishing  returns. 

Therefore,  the  goal  for  this  performance  objective  (>  15  percent  reduction  of  total  data  center 
energy)  was  met.  The  results  fell  just  short  of  the  stretch  goal  (>  20  percent  reduction). 

6.3  IMPROVED  COMPUTATIONAL  ENERGY  EFFICIENCY  [P03] 

Metric:  Compute  Rate  /  IT  Equipment  Power  (MFLOPS/watt) 

Goal:  Immersion  cooling  >  0  percent  improvement  compared  to  the  Base  Case 

Stretch  Goal:  Immersion  cooling  10  percent  improvement  compared  to  the  Base  Case 

This  performance  objective  is  intended  to  investigate  to  what  extent,  if  any,  the  immersion 
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cooling  alters  the  IT  equipment  computational  efficiency.  The  computational  efficiency  is 
defined  as:  compute  performance  (teraflops,  or  TFs)  divided  by  the  IT  equipment  power  (kW). 

The  performance  information  resulting  from  LINPACK  runs  and  the  electrical  power  consumed 
by  the  IT  equipment  were  measured  during  a  limited  test  period  during  the  Base  Case  and  the 
Pilot  Test  (immersion  cooling).  The  four  measurements  (two  power  and  two  performance)  were 
used  to  calculate  the  metric  for  this  performance  objective.  The  final  results  used  LINPACK 
parameters:  Problem  Size  (N)  =  672000,  Performance  Mode  on,  and  Turbo-Enabled. 

The  calculated  computational  energy  efficiency  for  the  Base  Case  was  857  MFLOPS/watt  versus 
809  MFLOPS/watt  for  the  Pilot  Case  (Table  6.3-1).  The  Pilot  Case  (immersion  cooling)  was 
more  than  5  percent  lower  than  the  Base  Case,  with  a  small  variance  (0.2  percent)  compared  to 
another  test  using  N=336000  (Figure  6.3-1).  Considering  the  small  variance,  we  conclude  it  is 
unlikely  the  goal  of  >  0  percent  would  be  met,  and  therefore  even  more  unlikely  the  stretch  goal 
would  be  met. 

Lower  energy  efficiency  for  the  Pilot  Test  (immersion  cooling)  is  likely  caused  by  the  higher 
CPU  temperatures  compared  to  the  Base  Case  (direct  liquid  cooling).  The  goal  is  not  likely 
achievable  with  the  high  boiling  temperature  of  Novec  649. 


Table  6.3-1:  Compute  Performance,  IT  Equipment  Power,  and  Computational  Energy 

Efficiency 


Test 

Compute 

Performance 

(TFs) 

IT  Equipment 
Power 
(kW) 

Metric 

(MFLOPS/watt) 

Improvement 

Base  Case 

24.33 

28.404 

857 

NA 

Pilot  Case 

23.93 

29.58 

809 

-5.6% 

Table  Note:  TFs 

=  teraflops  =  1012i 

loating  point  operations  per  second 

MFLOPS  =  megaflops  =  106  floating  point  operations  per  second 
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Base  Case  vs.  Pilot  Immersion  Cooling  Compute  Efficiency  Results 
Software:  High  Performance  UNPACK 

Base  Case  SGI  Cooling  (  Nov.  2013)  Pilot  Tests  Immersed  Novec  649  Cooling  (Sept.  2014) 


N=336000  N=672000  N=336000  N=336000  N=504000  N=672000  N=336000  N=504000  N=672000 

Performance  Performance  Performance  Demand  Performance  Performance  Performance  Performance  Performance 

Turbo-Enabled  Turbo-Enabled  Turbo-Enabled  Turbo-Enabled  Turbo-Enabled  Turbo-Enabled  Turbo-Disabled  Turbo-Disabled  Turbo-Disabled 


N  =  UNPACK  Problem  Size 

Figure  6.3-1:  Other  Computational  Energy  Efficiency  Results 

6.4  LOW  CONCENTRATION  OF  NOVEC  649  VAPORS  DURING  NORMAL 
OPERATION  [P04] 

Metric:  Number  of  8-hour  time-weighted  average  (TWA)  periods  with  concentrations 

above  the  goal  or  stretch  goal 

Goal:  Zero  periods  above  150  ppmV 

Stretch  Goal:  Zero  periods  above  75  ppmV 

This  performance  objective  was  evaluated  using  the  8-hour  time-weighted  average  (TWA) 
exposure  methodology.  From  the  SDS  for  Novec  649  (SDS  2015a)  the  TWA  concentration  limit 
is  150  ppmV.  The  goal  was  to  have  zero  8-hour  TWA  periods  above  150  ppmV.  The  stretch  goal 
was  zero  periods  above  75  ppmV.  The  vapor  level  was  measured  12  times  per  hour.  The  vapor 
measurement  period  started  October  7,  2014,  and  ended  August  5,  2015.  Since  the  first  10  days 
were  deemed  to  be  the  startup  period,  the  normal  operation  period  was  October  17,  2014  to 
August  5,  2015. 

Novec  649  vapor  concentrations  were  measured  in  two  places:  Above  the  tank  lid  at  what  was 
termed  the  operator’s  breathing  zone  and  under  the  raised  access  floor  adjacent  to  the  tank.  Since 
Novec  649  is  heavier  than  air,  there  was  a  concern  that  high  levels  under  the  floor  may  spread  to 
other  areas  inside  the  data  center.  The  underfloor  measurements  were  always  lower  than  the 
measurements  at  the  breathing  zone  (Figure  6.4-1),  and  were  lower  than  harmful  levels. 
Therefore,  the  TWA  values  for  underfloor  measurements  were  not  considered  important,  and  the 
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breathing  zone  measurements  were  used  to  calculate  the  TWA  values  for  this  performance 
objective. 

The  measurements  (bottom  graph)  and  calculated  8-hour  TWA  vapor  concentrations  (top  graph) 
are  shown  in  Figure  6.4-2.  The  maximum  TWA  value  was  48  ppmV,  therefore  both  the  goal  and 
stretch  goal  were  met. 


NRL  Immersion  Demonstration:  Novec  649  Vapor  Level  Data 
Day  10  [10/17/2014]  to  Day  300  [8/5/2015] 
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Figure  6.4-1:  Breathing  Zone  Vapor  Concentration  Compared  to  Floor  Zone  Vapor 

Concentration  During  Normal  Operation 
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NRL  Immersion  Demonstration:  Novec  649  Vapor  Level  Data:  Rolling  8  Hour  Average  &  Measurements 

Normal  Operation 

Day  10  [10/17/2014]  to  Day  300  [8/5/2015] 
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Figure  6.4-2:  Breathing  Zone  Novec  649  Vapor  Measurements  and  Calculated  8-Hour 
Time  Weighted  Averages  for  Normal  Operation 
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6.5  LOW  CONCENTRATION  OF  NOVEC  649  VAPORS  DURING  STARTUP 
OR  MAINTENANCE  [P05] 

Metric:  Maximum  measured  vapor  concentration 

Goal:  Zero  measurements  above  100,000  ppmV 

This  performance  objective  was  created  to  capture  incidents  when  operators  may  be  exposed  to 
high  vapor  concentration  levels  for  short  periods  of  time  while  performing  startup  or 
maintenance  activities  on  the  bath  or  IT  equipment  in  the  bath. 

We  assumed  that  the  no  observed  adverse  effect  limit  (NOAEL)  applies.  The  NOAEL  for  Novec 
649  vapor  is  100,000  ppmV,  based  on  a  cardiac  sensitization  study  and  a  four-hour  acute 
inhalation  study  (SDS  2015a).  This  evaluation  assumes  startup  and  maintenance  incidents  would 
last  less  than  four  continuous  hours. 

As  mentioned  in  Section  6.4,  the  Novec  649  vapor  underfloor  concentrations  were  consistently 
much  lower  than  those  observed  at  the  breathing  zone,  so  this  analysis  covers  only  the 
measurements  at  the  breathing  zone.  The  period  from  October  17,  2014,  to  March  5,  2015,  had 
zero  breathing  zone  vapor  concentrations  above  200  ppmV.  Consequently,  this  period  is  within 
the  NOAEL. 

There  was  a  brief  period  for  a  single  week,  starting  from  the  time  the  tank  was  filled  on  October 
7,  2014,  when  the  vapor  monitoring  system  was  sending  improperly  formatted  concentration 
values  to  the  database.  We  were  able  to  re-interpret  the  database  values  as  plotted  in 
Figure  6.5-1.  The  maximum  corrected  concentration  measurement  was  approximately 
120  ppmV. 

Our  corrected  data  indicate  zero  measurements  above  200  ppmV  (Figure  6.5-1  green  trace)  for 
the  entire  vapor  concentration  evaluation  period.  Therefore  the  goal  (<  100,000  ppmV)  for  this 
performance  objective  was  achieved. 
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Figure  6.5-1:  Startup  Period  Breathing  Zone  Novec  649  Vapor  Concentration 
Measurements.  Vertical  axis  is  in  units  of  ppmV. 
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6.6  REDUCTION  IN  GREENHOUSE  GAS  EMISSIONS  [P06] 

Metric:  Carbon  Dioxide  Equivalent  (CDE)  reduction  using  the  EPA  Greenhouse  Gas 

Equivalencies  Calculator 

Goal:  Lower  CDE  compared  to  the  Base  Case 

The  calculator  uses  the  U.S.  annual  non-baseload  CO2  output  emission  rate  in  the  Emissions  & 
Generation  Resource  Integrated  Database  (eGRID)  to  convert  reductions  of  kilowatt-hours  into 
avoided  units  of  carbon  dioxide  emissions.  We  used  metric  tons  as  the  unit  of  measure.  The 
conversion  is  6.89551  x  10‘4  metric  tons  C02/kWh  (EPA  2015). 

There  are  at  least  two  perspectives  that  can  be  applied  to  analyzing  the  direct  greenhouse  gas 
reductions:  (1)  to  base  the  reduction  on  the  overall  data  center  energy  savings,  or  (2)  to  base  the 
reduction  on  the  data  center  cooling  energy  savings  only.  The  results  of  both  perspectives  are 
presented  below. 

Three  different  immersion  cooling  alternatives,  along  with  the  Base  Case,  were  compared  for 
two  data  center  efficiencies  (low  and  high).  The  three  immersion  cooling  alternatives  are 
described  in  Section  6.1  and  pictured  in  Appendix  C. 

The  values  for  yearly  energy  use  (overall  and  cooling)  come  from  the  2  MW  IT  design  load 
simulation  results  as  described  in  Section  6.1  and  Appendix  C.  The  eight  (8)  resulting 
combinations  were  used  to  create  the  six  (6)  reduction  comparisons  in  Table  6.1-2. 

The  CDE  metric  ton  reduction  was  determined  by  entering  the  yearly  energy  use  for  each  of  the 
sixteen  cases  using  the  online  EPA  calculator  (EPA  2015). 

It  was  assumed  that  a  DoD  data  center  would  most  likely  to  be  faced  with  the  choice  of  selecting 
between  the  Base  Case  and  Immersion  Case  in  a  low-efficiency  data  center.  Immersion  Case  3A 
(with  cooling  tower)  was  assumed  to  be  the  most  likely  cooling  infrastructure  selected  if  the 
decision  was  to  choose  immersion  cooling.  Therefore,  Base  Case  A  and  Immersion  Case  3A 
were  used  to  calculate  the  metric  value  for  this  performance  objective  (Table  6.6-1). 

Since  changes  to  one  data  center  system  can  affect  other  systems,  looking  at  the  overall  data 
center  energy  use  is  the  most  accurate. 

The  CDE  greenhouse  gas  emissions  were  19  percent  lower  when  Immersion  3 A  cooling  was 
selected,  compared  to  Base  Case  A  (hybrid)  cooling  based  on  the  total  data  center  energy 
(Table  6.6-1).  It  is  interesting  to  note  that  the  CDE  tons  saved  based  on  the  total  overall  data 
center  energy  (2,772  tons)  were  very  close  to  those  for  cooling  energy  (2,744  tons)  (Table  6.6-2). 
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This  suggests  that  the  savings  for  cooling  was  responsible  for  virtually  all  of  the  energy  savings. 
This  is  expected  because  the  changes  were  to  the  cooling  system  only. 


Table  6.6-1:  CDE  Changes  Based  on  Total  Data  Center  Energy 


Low  Efficiency  Data  Center 


Data  Center 
Electricity 
Consumed 
(kWh) 

Carbon  Dioxide 
Equivalent 
(metric  tons) 

Carbon  Dioxide 
Equivalent 
Reduction 
(metric  tons) 

Carbon  Dioxide 
Equivalent 
Reduction 

(%) 

Basecase  A 

20,848,863 

14,376 

Immersion  1A 

20,302,681 

14,000 

376 

3% 

Immersion  2A 

16,926,384 

11,672 

2,704 

19% 

Immersion  3A 

16,829,028 

11,604 

2,772 

19% 

High  Efficiency  Data  Center 


Basecase  B 

16,999,529 

11,722 

Immersion  IB 

16,599,191 

11,446 

276 

2% 

Immersion  2B 

16,177,945 

566 

4% 

Immersion  3B 

16,031,990 

11,055 

667 

5% 

Table  6.6-2:  CDE  Changes  Based  on  Cooling  Energy 


Low  Efficiency  Data  Center 


Cooling 

Electricity 

Consumed 

(kWh) 

Carbon  Dioxide 
Equivalent 
(metric  tons) 

Carbon  Dioxide 
Equivalent 
Reduction 
(metric  tons) 

Carbon  Dioxide 
Equivalent 
Reduction 

(%) 

Basecase  A 

5,476,527 

3,776 

Immersion  1A 

4,936,154 

3,404 

372 

10% 

Immersion  2A 

1,592,441 

1,098 

2,678 

71% 

Immersion  3A 

1,495,949 

1,032 

2,744 

73% 

High  Efficiency  Data  Center 


Basecase  B 

1,788,225 

1,233 

Immersion  IB 

1,391,368 

959 

274 

22% 

Immersion  2B 

973,903 

672 

561 

45% 

Immersion  3B 

829,211 

572 

661 

54% 
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6.7  DIELECTRIC  LIQUID  LOSS  [P07] 

Metric:  Cost  ($)  Novec  649  /  Cost  ($)  IT  Electrical  Energy 

Goal:  Less  than  or  equal  to  1  percent 

Stretch  Goal:  Less  than  0. 1  percent 

The  Novec  649  is  expensive  at  $75/liter  compared  to  “free”  liquids  such  as  air  and  water,  which 
are  typically  used  for  heat  transfer  in  data  center  applications.  Novec  649  is  very  volatile,  and  if 
left  in  an  unsealed  container  will  evaporate  at  a  considerable  rate. 

This  performance  objective  is  targeted  to  find  if  the  equipment  demonstrated  does  an  adequate 
job  of  containing  Novec  649.  The  goal  of  1  percent  and  stretch  goal  of  0.1  percent  were 
considered  reasonable  at  the  time  the  Demonstration  Plan  was  developed.  However,  the  liquid 
loss  turned  out  to  be  much  higher  than  the  goal. 

The  metric  for  this  PO  was  evaluated  over  the  time  period  starting  on  10/14/2014  and  ending 
on  8/5/2015.  The  liquid  lost  and  IT  electrical  energy  consumed  during  this  period  are 
estimated  below. 

Liquid  Lost 

Two  quantities  are  needed  to  estimate  the  liquid  lost  during  the  period: 

1.  Amount  of  liquid  added  to  the  bath 

2.  The  amount  of  liquid  needed  to  bring  the  final  liquid  level  to  the  level  at  the  start  of  the 
evaluation  period 

Liquid  Added  to  the  Bath 

The  Novec  649  liquid  required  to  make  up  for  losses  was  provided  in  12-liter  bottles. 
These  bottles  contained  17.2  kilograms  (kg)  of  liquid  each  (10.64  liters  at  20°C).  The 
liquid  was  added  as  needed  during  the  demonstration  to  keep  the  IT  equipment  covered 
with  the  Novec  649  liquid.  The  total  amount  of  liquid  added  was  determined  by  obtaining 
a  count  of  bottles  emptied  into  the  bath  during  the  evaluation  period.  A  total  of  six  bottles 
were  emptied  into  the  bath  (J.  Osbum,  NRL,  pers.  comm.  2015).  Therefore,  the  known 
direct  additions  totaled  103.2  kg  (63.85  liters  at  20°C). 

The  Amount  of  Liquid  Needed  to  Bring  the  Final  Liquid  Level  to  the  Original  Level 

The  level  of  the  liquid  in  the  bath  was  continuously  measured  and  recorded  in  the  PI 
database.  The  database  values  (count)  at  the  end  and  beginning  of  the  evaluation  period 
were  1927  and  2139,  respectively  (Figure  6.7-1)  at  50°C.  The  translation  of  level  values 
(count)  found  in  the  database  to  a  volume  change  in  the  bath  is  8.68  per  liter 
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(Appendix  B).  Therefore  212  (2139  -  1927)  level  value  change  equals  24.42  liters  at 
50°C  or  23.10  liters  at  20°C. 

Liquid  Loss  Summary 

The  total  liquid  lost  was  87  (63.85+23.10)  liters.  The  dollar  value  of  the  liquid  lost  was 
$6,525  (87  liters  x  $75/liter). 

IT  Equipment  Electrical  Energy  Consumed 

The  electrical  power  to  the  IT  equipment  in  the  bath  was  separately  metered  and  the  data  were 
continuously  recorded  in  the  PI  database.  The  metered  data  included  accumulated  kilowatt- 
hours,  and  therefore  the  total  electrical  energy  could  be  determined  by  subtracting  the  reading  of 
10/14/2104  from  that  on  8/5/2015.  The  difference  of  these  two  readings  was  19,667  kWh.  The 
cost  of  electrical  power  at  the  demonstration  site  was  $0.09/kWh  (J.  Osburn,  NRL,  pers.  comm. 
2015).  Therefore,  the  cost  of  the  electrical  energy  consumed  by  the  IT  equipment  during  the 
liquid-loss  evaluation  period  was  $1,770. 

Summary 

The  cost  of  the  lost  liquid  and  electrical  energy  consumed  were  $6,525  and  $1,770,  respectively, 
with  a  ratio  of  3.68,  or  368  percent.  The  goal  and  stretch  goal  of  1  percent  and  0.1  percent, 
respectively,  were  not  met. 

A  number  of  potential  contributing  factors  causing  the  unacceptable  and  unexpected  liquid  loss 
were  discussed.  One  factor  could  be  the  time  the  bath  lid  was  open,  which  was  more  than 
expected  due  to  the  high  number  of  maintenance  requests  (Section  6.12).  In  addition,  turbulent 
airflow  within  the  data  center  may  have  contributed  to  the  high  loss  rate.  There  will  be  technical 
challenges  containing  volatile  fluids.  Experiments  attempting  to  characterize  and  isolate  the 
liquid  loss  mechanisms  were  not  conducted. 
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NRL  Immersion  Demonstration:  Tank  Fluid  Level  Data 
Day  7  [10/14/2014]  to  Day  301  [8/5/2015] 
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Figure  6.7-1:  Liquid  Level  and  Temperature  Measurement  Data 


6.8  SYSTEM  ECONOMICS  (QUALITATIVE)  [P08] 

Metric:  Simple  payback  period 

Goal:  Payback  less  than  4  years 

Stretch  Goal:  Payback  less  than  3  years 

The  economic  analysis  assumed  that  a  low-efficiency  infrastructure  data  center  was  scheduled  to 
replace  all  IT  equipment  with  supercomputing  equipment,  and  there  were  two  choices  for  the 
replacement  cooling  infrastructure: 

1.  Hybrid  (water  and  air)  direct  cooling  (Base  Case  A) 

2.  Two-phase  immersion  cooling  (Immersion  Case  3A) 

Both  cases  assumed  an  average  IT  power  level  of  909  kW.  This  power  level  equated  to  16  SGI 
M-Racks  and  32  immersion  baths.  The  input  assumptions  for  calculating  the  payback  period  for 
Immersion  Case  3A  are  provided  in  Table  6.8-1  (Forecast  Data). 

The  economic  analysis  is  a  forecast  based  on  improved  metrics  compared  to  the  demonstration 
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results.  For  example,  based  on  measurements,  the  estimate  for  the  liquid  loss  was  much  higher 
than  expected  (104  liters/year).  The  technology  would  not  be  viable  without  a  dramatically  lower 
liquid  loss.  The  liquid  loss  used  in  the  forecast  was  assumed  to  be  10  percent  of  the  cost  of  the 
electric  energy  for  the  IT  equipment,  or  18  liters  per  year  per  bath.  Table  6.8-1  shows  the 
demonstration  results  and  other  forecasted  parameters  (reduced  by  50  percent)  used  in  the  NPV 
and  payback  calculations. 


Table  6.8-1:  Immersion  Case  Results  and  Forecasted  Data 


Metric 

Immersion 

Results 

Forecast 

Data 

85  liters  for  10  months 
($6,525) 

1 8  liters/year 
($1,350) 

Novec  649  Liquid  Loss 

102  liters/year 
($7,650) 

Novec  649  Bath 

595  liters 

297  liters 

Liquid  Volume 

($44,625) 

($22,312) 

Bath  Cost 

$91,000  each 

$45,500  each 

The  system  economics,  comparing  one  case  to  the  other  (Base  Case  A  and  Immersion  Case  3A), 
were  evaluated  using  two  financial  measures: 

1.  Calculate  the  simple  payback  period 

2.  Calculate  the  NPV  using  the  NIST  building  life  cycle  cost  (BLCC)  calculator 

Note  that  the  comparison  is  between  Base  Case  A  and  Immersion  Case  3A  with  the  forecast  data 
in  Table  6.8-1. 

Three  inputs  were  used: 

•  First-year  capital  costs  (IT  equipment  and  cooling  equipment) 

•  Yearly  energy  consumed  for  the  entire  data  center  (based  on  simulated  PUE  results) 

•  Yearly  cost  of  3M  Novec  649  liquid  losses  (immersion  case  only) 

Payback  Period  Analysis 

The  first-year  capital  costs  for  the  Immersion  Case  were  $1.9  million  higher  compared  to  the 
Base  Case  (Figure  6.8-1).  The  Immersion  Case  shows  a  $103K/year  reduction  in  energy  use  and 
a  liquid  loss  cost  of  $43.1K/year,  resulting  in  a  net  operational  cost  reduction  of  $60K/year.  The 
simple  payback  period  is  therefore  approximately  32  years  ($1.9  million/$60K).  The  goal 
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(<  4  years)  and  stretch  goal  (<  3  years)  were  not  met. 


Net  Present  Value  Analysis 

A  seven-year  life  cycle  cost  comparison  was  calculated  for  the  two  choices  using  the  NIST 
BLCC  calculator.  The  input  assumptions  for  the  calculator  are  shown  in  Figure  6.8-1,  and  the 
results  from  the  analysis  are  provided  in  Figure  6.8-2. 

The  NPV  for  the  Immersion  Case  was  calculated  to  be  9.5  percent  more  than  the  Base  Case.  The 
higher  NPV  is  primarily  due  to  the  cost  of  initially  filling  the  baths  with  Novec  649 
($22.312k/bath  for  32  baths,  for  a  total  of  $7 13.98k)  and  bath  enclosure  costs  ($45.5k/bath  for  32 
baths,  for  a  total  of  $  1,456k).  These  results  suggest  that  design  improvements,  beyond  those 
assumed  for  this  analysis  (see  Forecast  Data  in  Table  6.8-1),  are  needed  before  the  immersion 
cooling  option  will  be  financially  viable.  The  higher  capital  cost  for  the  immersion  cooling 
option  may  be  offset  by  lower  cooling  infrastructure  capital  costs  if  a  new  data  center  is 
involved. 

Summary 

The  initial  fill  volume  and  bath  cost  were  high  because  the  IT  equipment  used  in  the 
demonstration  was  not  specifically  designed  for  two-phase  immersion  cooling.  Before 
immersion  cooling  can  be  cost  competitive  with  existing  cooling  methods  there  needs  to  be  a 
substantial  increase  in  the  amount  of  IT  equipment  that  can  be  contained  in  a  given  volume.  This 
density  increase  may  involve  a  complete  rethinking  of  current  HPC  computing  architecture. 
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Version:Oct.  21,  2015 

Assumptions 


Total  Design  IT  Power 

908.79 

kW 

Maximum  IT  Power  per  Bath 

28.4 

kW 

Design  Number  of  Baths 

32.00 

Maximum  IT  Power  per  M-Rack 

56.80 

kW 

Design  Number  of  M-Racks 

16.00 

CDU 

$ 

15,000 

/ea 

Cooling  Rack 

$ 

15,000 

/ea 

M-Rack 

$ 

3,000 

/ea 

IT  Hardware  per  M-Rack  w/o  CPUs 

$ 

500,000 

/M-Rack 

CPUs  per  M-Rack 

288 

CPU  Component  Cost 

$ 

1,000 

/ea 

BM  Novec649 

s 

_ Z5j 

/liter 

Novec  649  Initial  Fill  per  Bath 

297 

liters 

Cost  of  Bath  Enclosure 

45,500 

ea 

Fluid  Loss  per  Year  per  Bath 

17.95 

liters 

Operational  Average  IT  Power  per  M-Rack 

28.40 

kW 

Operational  Average  IT  Power  per  Bath 

14.20 

kW 

Cost  of  Electricity 

0.090 

/kWh 

PUE  for  Base  Case 

1.491 

PUE  for  Immersion  Case 

1.203 

Year  1 

Year  2 

Year  3 

Year  4 

Year5 

Year  6 

Year  7 

Base  Case 

Capital  Cost  (16  M-Racks) 

CDUs  (4) 

$ 

60,000 

Cooling  Racks  (8) 

$ 

120,000 

M-Racks (16) 

$ 

48,000 

IT  Hardware  w/o  CPUs  (16  M-Racks) 

s 

8,000,000 

CPU  Components  (288  x  16  =  4608) 

$ 

4,608,000 

Total 

$ 

12,836,000 

Operation  Cost  (16  M-Racks) 

Electrical  Power(454.4  kW,  PUE  1.491)  kWh 

5,934,991 

5,934,991 

5,934,991 

5,934,991 

5,934,991 

5,934,991 

5,934,991 

Electrical  Energy  Cost 

$ 

534,149 

$  534,149 

$  534,149 

$  534,149 

$  534,149 

$  534,149 

$  534,149 

Total 

$ 

13,370,149 

$534,149 

$534,149 

$534,149 

$534,149 

$534,149 

$534,149 

Immersion  Case 

Capital  Cost  (32  Baths) 

Baths(32) 

$ 

1,456,000 

3M  649  (297  Iitersx32x$75) 

$ 

712,792 

IT  Hardware  w/o  CPUs  (16  M-Racks) 

$ 

8,000,000 

CPU  Components  (144  x  32  =  4608) 

$ 

4,608,000 

Total 

$ 

14,776,792 

Operational  Cost  (32  Baths) 

Electrical  Energy  (454.4  kW,  PUE  1.22)  kWh 

4,788,594 

4,788,594 

4,788,594 

4,788,594 

4,788,594 

4,788,594 

4,788,594 

Electrical  Energy  Cost 

$ 

430,973 

$  430,973 

$  430,973 

$  430,973 

$  430,973 

$  430,973 

$  430,973 

Fluid  Loss(32*104*75)$ 

$ 

43,080 

$  43,080 

$  43,080 

$  43,080 

$  43,080 

$  43,080 

$  43,080 

Total 

$ 

15,207,766 

$430,973 

$430,973 

$430,973 

$430,973 

$430,973 

$430,973 

Figure  6.8-1:  BLCC  Version  5.3-15  Inputs 
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NIST  BLCC  5.3-15:  Summary  LCC 

Consistent  with  Federal  Life  Cycle  Cost  Methodology  and 
Procedures,  10  CFR,  Part  436,  Subpart  A 


General  Information 


File  Name: 

Life-Cycle-Cost-v7 

Date  of  Study: 

Wed  Oct  21 10:46:35  PDT  2015 

Analysis  Type: 

FEMP  Analysis,  Energy  Project 

Project  Name: 

Immersion 

Project  Location: 

District  of  Columbia 

Analyst: 

LBNL 

Base  Date: 

l-Jan-16 

Service  Date: 

l-Jan-16 

Study  Period: 

7  years  0  months  (January  1,  2016  through 
December  31, 2022) 

Discount  Rate: 

3% 

Discounting  Convention: 

End-of-Year 

Discount  and  Escalation  Rates  are  REAL 
(exclusive  of  general  inflation) 


Alternative:  Basecase 

LCC  Summary 


Present  Value 

Annual  Value 

Initial  Cost 

$12,836,000 

$2,060,796 

Energy  Consumption 

$3,412,326 

$547,842 

Energy  Demand  Costs 

$0 

$0 

Energy  Utility  Rebates 

$0 

$0 

Water  Usage  Costs 

$0 

$0 

Water  Disposal  Costs 

$0 

$0 

Annually  Recurring 

$0 

$0 

Non-Annually  Recurring 

$0 

$0 

Replacement  Costs 

$0 

Less  Remaining  Value 

$0 

$0 

Total  Life-Cycle  Cost 

$16,248,326 

$2,608,638 

Alternative:  Immersion 

LCC  Summary 


Present  Value 

Annual  Value 

Initial  Cost 

$14,776,792 

$2,372,386 

Energy  Consumption 

$442,022 

Energy  Demand  Costs 

$0 

Energy  Utility  Rebates 

$0 

$0 

Water  Usage  Costs 

$0 

$0 

Water  Disposal  Costs 

$0 

$0 

Annually  Recurring 

$268,416 

Non-Annually  Recurring 

$0 

Replacement  Costs 

$0 

$0 

Less  Remaining  Value 

$0 

$0 

— 

|Total  Life-Cycle  Cost 

$17,798,412 

$2,857,501 

Figure  6.8-2:  BLCC  Version  5.3-15  Calculation  Summary 
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6.9  LOWER  CPU  CHIP  TEMPERATURES  [P09] 


Metric:  Average  CPU  Temperature 

Goal:  Equal  or  below  the  Base  Case 

To  find  out  if  the  immersion  cooling  technology  affected  the  CPU  chip  temperatures  compared 
the  Base  Case,  the  CPU  temperature  data  was  collected  during  the  Base  Case  test  and  Pilot  Test. 
To  provide  a  consistent  IT  equipment  computing  load,  the  software  benchmarking  tool 
LINPACK  was  run  during  both  tests. 

The  Base  Case  test  temperature  data  for  all  144  CPUs  were  obtained  over  approximately  a 
one-hour  period  (Figure  6.9-1).  Dashed  lines  for  two  temperature  traces  (high  temperature  and 
low  temperature)  are  shown  to  point  out  that  individual  CPUs  operated  a  bit  warmer  or  cooler 
than  other  CPUs.  The  reasons  for  the  temperature  difference  across  the  144  CPUs  were  not 
investigated.  The  mean  temperature  for  the  Base  Case  test  was  51.6°C. 


ICE  X  CPU  Temperatures:  SGI  Base  Case  Run,  UNPACK  Load 

Data  From  Nov  12, 2013  (SGI  Chippewa  Falls  MN)  -  144  CPUs 
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Figure  6.9-1:  Base  Case  CPU  Temperatures 
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The  Immersion  Case  CPU  temperature  data  was  obtained  on  September  9,  2014  during  the  Pilot 
Test.  The  mean  temperature  during  the  Immersion  Case  test  between  the  hours  of  18:30  and 
20:00  was  70.2°C.  The  low  temperature  periods  (Figure  6.9-2)  are  caused  by  the  UNPACK 
software  restarting  after  solving  a  set  of  linear  equations.  As  expected,  during  these  low -power 
periods  the  CPU  temperature  approached  the  Novec  649  boiling-point  temperature  of  49°C. 

A  small  number  of  CPUs  reported  temperatures  considerably  higher  than  75°C  during  the 
Immersion  Case  run  (Figure  6.9-2).  The  cause  of  this  was  not  investigated.  3M  has  suggested 
that  these  high  temperatures  may  be  caused  by  contamination  of  the  boiling-enhancement 
coating  on  the  CPUs  and  thereby  decreasing  the  heat  flux. 

The  goal  (equal  or  below  the  Base  Case)  for  this  performance  objective  was  not  met.  The  CPU 
temperatures  averaged  approximately  20°C  higher  when  the  computer  was  immersion-cooled 
compared  to  the  Base  Case.  This  higher  temperature  may  have  been  due  to  a  couple  of 
contributing  factors.  The  liquid  temperature  close  to  the  CPU  is  49C  in  the  immersion  case 
(Novec  649  boils  at  49C)  and  20C  in  the  Base  Case  (20C  cooling  water).  The  goal  is  not  likely 
achievable  with  the  high  boiling  temperature  of  Novec  649.  The  other  contributing  factor  is  that 
the  phase  change  taking  place  on  chip  heat-transfer  surfaces  may  also  have  deposited  pollutants, 
which,  in  turn,  would  have  limited  the  heat  transfer. 

ICE  X  CPU  Temperatures:  3M  649  Immersion  Pilot  Run,  UNPACK  Load 

Test  Data  Sept  9,  2014  (SGI  Chippewa  Falls  MN)  -  144  CPUs 
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Figure  6.9-2:  Immersion  Case  CPU  Temperatures 
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6.10  HIGH  USER  SATISFACTION,  LOW  NUMBER  OF  CONCERNS 
(QUALITATIVE)  [PO10] 

The  users  (NRL  personnel)  were  asked  to  report  their  input  on  two  categories  of  concerns: 

(1)  safety  and  (2)  operational.  For  each  category,  the  user  was  asked  to  list  concerns  and 

concerns  that  were  not  resolved. 

Metric:  Number  of  safety  concerns 

Number  of  safety  concerns  that  were  not  resolved 

Number  of  operational  concerns 

Number  of  operational  concerns  that  were  not  resolved 

Goal:  Zero  unresolved  safety  concerns 

Stretch  Goal:  The  goal  met  +  zero  operational  concerns  that  were  not  resolved 

Safety  Concerns 

1.  Immersed  electronics  failed  during  the  demonstration.  There  was  a  concern  that 
perfluoroisobutylene  (PFIB)  formed  from  spark  events  may  have  been  involved  in  the 
failures.  3M  tested  the  liquid  for  PFIB  contamination  and  none  was  found.  Naval 
Research  Laboratory  personnel  did  not  have  a  concern  regarding  PFIB  exposure  after  the 
3M  test  results  were  known. 

2.  Using  gloves  while  servicing  equipment  in  the  bath  was  recommended.  Service  personnel 
reported  that  gloves  did  not  allow  the  needed  dexterity.  Therefore,  service  was  often 
performed  without  gloves  although  it  caused  skin  dryness.  The  NRL  did  not  consider  this 
to  be  a  safety  concern  based  on  the  SDS  sheet  for  Novec  649  (“no  protective  gloves 
required”). 

The  NRL  personnel  reported  that  both  safety  concerns  were  resolved.  Therefore,  the  goal 

was  met. 

Operational  Concerns 

NRL  personnel  reported  14  Operational  Concerns: 

1.  Glass  lid  susceptible  to  cracking 

2.  Glass  lid  unwieldy 

3.  No  status  indication  on  filters  (carbon,  desiccant,  and  silica  gel) 

4.  Service  near  the  bottom  of  the  bath  was  difficult 

5.  Working  in  the  bath  with  gloves  often  resulted  in  poor  dexterity 

6.  Bath  documentation  was  nonexistent 


62 


7.  Novec  649  vapor  bubbles  degraded  the  visibility  during  service 

8.  Power  supplies  failed  early  in  the  demonstration 

9.  The  apparatus  used  for  filling  the  bath  initially  caused  loss  of  liquid 

10.  Liquid  loss  during  the  demonstration 

1 1 .  Unable  to  tell  if  the  top  lid  was  sealed 

12.  Curtain  around  the  bath  (to  limit  the  vapor  loss)  had  to  be  pushed  aside 
during  maintenance 

13.  High  number  of  logic  board  failures 

14.  InfiniBand  (networking  boards)  connectivity  not  consistent 

The  goal  was  met,  but  the  stretch  goal  was  not. 

Other  than  the  repeated  electronic  failures  (Concerns  13  and  14),  overcoming  the  remaining 
operational  concerns  could  also  be  a  major  technical  challenge. 
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6.11  IMPROVED  IT  POWER  DENSITY  (QUALITATIVE)  [POll] 


Metric:  Compute  power  per  square  foot  data  center  floor  space  (kW/ft2) 

Goal:  Equal  to  or  better  than  the  Base  Case 

Stretch  Goal:  Significantly  better  than  the  Base  Case 

Heat  or  power  density,  IT  power  per  square  foot  data  center  floor  space  (kW/ft2),  is  an  attribute 
cited  as  a  potential  advantage  with  two-phase  immersion  cooling.  The  actual  metric  used  for  this 
performance  objective  is  sockets  per  square  foot  data  center  floor  space  (sockets/ft2)  as  a 
substitute  for  kW/ft2.  Supercomputer  manufacturers  commonly  use  the  sockets/ft2  metric  as  a 
measure  of  density.  Each  socket  can  hold  one  CPU.  The  computing  throughput  and  maximum 
power  consumption  is  largely  determined  by  the  CPU  make  and  model. 

Base  Case  Density  (hybrid  cooling) 

Figure  6.11-1  shows  the  highest  density  (sockets/ft2)  using  the  SGI  ICE  X  model  equipped  with 
the  cooling  option  used  in  the  Base  Case.  Cooling  distribution  units  are  included  in  the  required 
space  because  they  are  a  required  component  of  the  SGI  ICE  X  system.  Floor  space  for  aisle 
ways  is  also  included.  The  density  (sockets/ft2)  for  the  SGI  ICE  X  (Base  Case)  maximum  density 
layout  is  9.44  sockets  per  square  foot. 

Immersion  Case  Density 

Figure  6.11-2  shows  an  immersion  cooling  multi-tank  layout  using  the  dimensions  of  the  NRL 
demonstration  bath.  Aisle  ways  are  added  to  enable  installation  or  removal  of  baths.  The  density 
for  the  Immersion  Case  was  2.1  sockets  per  square  foot.  Space  that  may  be  saved  by  removing 
cooling  equipment  from  the  data  hall  was  not  considered. 

The  Immersion  Case  had  a  density  of  only  22  percent  (2.1/9.44)  of  the  Base  Case.  Thus,  neither 
the  goal  or  the  stretch  goal  for  this  performance  objective  was  met. 

The  results  for  this  performance  objective  were  expected.  The  demonstrated  IT  equipment  was 
designed  to  be  air  cooled  and  therefore  was  not  optimized  for  two-phase  immersion  cooling.  In 
addition,  the  bath  was  a  prototype  design.  Future  bath  designs  could  significantly  improve  the 
maximum  density.  An  advantage  of  typical  air-cooled  IT  equipment  is  the  ability  to  stack  the 
equipment  vertically  using  the  common  6.5  foot  (2  m)-high  data  center  rack.  Immersion-cooled 
IT  equipment  is  currently  laid  out  horizontally  to  be  able  to  lower  the  equipment  into  the  bath. 
Consequently,  this  configuration  takes  up  more  floor  space  for  a  given  amount  of  IT  equipment 
compared  to  the  common  vertical  data  center  rack. 
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Figure  6.11-1:  Base  Case  SGI  ICE  X  M-Cell  Maximum  Density  Layout 
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Figure  6.11-2:  Immersion  Case  Layout  Using  Demonstrated  Dimensions 
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6.12  SYSTEM  MAINTENANCE  (QUALITATIVE)  [PQ12] 


Metric:  Number  of  IT  equipment  maintenance  requests  for  the  Immersion  Case  compared 

to  the  Base  Case  model 

Goal:  Maintenance  (labor  and  spares  costs)  required  not  considered  as  a  significant 

hindrance  to  adoption  of  this  technology 

Stretch  Goal:  Maintenance  required  not  greater  than  SGI  hardware  cooled  by  SGI's  commercial 
hybrid  cooling 

We  anticipated  that  the  maintenance  requests  for  immersion-cooled  IT  equipment  would  be  more 
frequent  compared  to  the  Base  Case. 

In  spite  of  anticipating  a  higher  number  of  maintenance  requests,  the  goal  was  that  the 
maintenance  would  not  pose  a  significant  hindrance  to  adopting  the  immersion  cooling 
technology.  The  stretch  goal  was  for  the  maintenance  request  frequency  to  be  similar  to  the  Base 
Case  type  of  IT  equipment. 

A  high  number  of  unexpected  power  supply  and  logic  board  failures  occurred  soon  after  starting 
the  demonstration.  The  resulting  quantity  of  opened  maintenance  cases  for  the  Immersion  Case 
were  much  higher  than  for  the  Base  Case  (Table  6.12-1).  The  IT  equipment  used  for  this 
immersion  demonstration  had  more  than  66  times  more  maintenance  cases  opened  than  the 
average  for  similar  IT  equipment.  Neither  the  goal  nor  the  stretch  goal  for  this  performance 
objective  was  met. 

The  cause  of  the  power  supply  failures  was  determined  and  a  subsequent  fix  was  successfully 
applied.  Although  considerable  resources  were  assigned  to  correct  the  cause  or  causes  of  the 
logic -board  failures,  the  efforts  were  unsuccessful.  A  large  number  of  metallic  filaments  "tin 
whiskers"  were  observed  on  failed  boards.  Although  the  exact  mechanism  for  creating  these  tin 
whiskers  is  unresolved,  they  are  suspected  of  creating  shorts  on  the  logic  boards.  Identifying  the 
root  cause(s)  and  a  solution  for  the  logic  board  failures  could  require  considerable  resources. 
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Table  6.12-1:  Immersion  Case  and  Base  Case  Maintenance  Freq 

uency 

Equipment  Description 

Number  of  Cases 
Opened 

Service  Hours 

Parts  Requested 

NRL  Immersion  Case 
(over  7  months) 

62 

235 

30 

Worldwide  Average  for 

144  Socket  Systems 
(last  12  months) 

1.6 

5.6 

1.5 

Percent  Increase 

6,643% 

4,096% 

1,900% 
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7.0  COST  ASSESSMENT 


The  demonstrated  OBI  cooling  technology  contains  a  number  of  additional  unique  costs 
compared  to  deploying  traditional  air-cooled  IT  equipment.  The  major  unique  costs  include  those 
for  the  Novec  649  liquid  and  a  custom  bath  enclosure. 

However,  two-phase  immersion  cooling  can  provide  opportunities  to  lower  particular  first  and 
operational  costs  that  may  offset  some  of  the  unique  additional  costs.  Two-phase  immersion 
cooling  can  be  supported  with  warm-water  cooling.  Depending  on  the  climate  and  whether  or  not 
the  installation  is  part  of  building  a  new  data  center  or  is  part  of  an  IT  equipment  update,  there 
may  be  an  opportunity  to  avoid  or  reduce  the  first  and  operational  costs  associated  with 
compressor-based  cooling  equipment. 

As  mentioned  previously,  the  cooling  water  temperature  needed  to  support  two-phase  OBI 
cooling  can  be  significantly  higher  than  is  typically  found  in  the  DoD’s  high-performance 
computing  data  center  buildings.  This  reduces  costs  because  low-first  cost  infrastructure 
technologies  such  as  dry  coolers  can  be  used,  rather  than  high-first  cost  technologies  such  as 
chillers.  In  most  climates,  a  dry  cooler  or  a  cooling  tower  dedicated  to  immersion  cooling  is 
adequate  to  provide  the  cooling  water  without  a  chiller.  Selecting  immersion-cooled  IT 
equipment  may  save  a  considerable  first  cost  funds  if  compressor-based  cooling  systems  can  be 
avoided  or  downsized. 

7.1  COST  MODEL 

Table  7-1  contains  cost  elements  from  the  demonstration  and  estimates  for  Bath  and  System 
Maintenance,  Hardware  Failures,  and  Operator  Training. 
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Table  7-1:  Cost  Elements 


Item 

Cost  Element 

Information  Tracked  During  the 
Demonstration 

Estimated 

Costs 

1 

Bath  Enclosure 

cost  of  bath  enclosure 

$91,000 

2 

Bath  and  Systems 
Installation 

piping,  cooling,  electrical,  sensors,  and 
data  collection  installation  costs  and 
estimates 

$100,000 

3 

3M  Novec  649  Liquid 

amount  of  liquid  used  for  initial  fill  and 
liquid  lost  from  10/2014  to  8/2015 

$44,550+ 

$6,525 

4 

Bath  and  System 
Maintenance 

estimates 

$400/yr. 

5 

Hardware  Failures 

glass  top  lid  cracked 

$500/yr. 

6 

Operator  Training 

estimates 

Initially  60  hrs. 

($3,000), 
Yearly  8  hrs. 
($400) 

Bath  Enclosure 

The  bath  assembly  is  the  primary  device  needed  to  implement  this  technology.  It  provides  the 
containment  for  the  immersion  liquid,  space  for  the  IT  equipment,  and  controls.  The  Technology 
Readiness  Level  (TRL)  is  a  common  gauge  of  how  far  along  the  technology  is  in  its  development 
cycle.  It  is  estimated  to  be  at  Level  7,  which  means  a  system  prototype  has  been  demonstrated  in 
an  operational  environment.  The  bath  enclosure,  purchased  from  3M,  cost  $91,000.  The  future 
cost  per  bath  will  depend  on  the  final  design  and  manufacturing  batch  sizes. 

Bath  and  Systems  Installation 

The  plumbing,  pump,  and  dry  cooler  costs  are  relevant  to  the  demonstration  set-up  only,  and 
should  not  be  used  to  estimate  the  costs  where  a  larger  number  of  immersion  baths  will  be 
deployed.  The  piping  and  main  cooling  system  elements  were  installed  by  a  plumbing  contractor 
hired  by  NRL.  The  fee  paid  to  the  contractor  was  $50,000.  The  pump  and  dry  cooler  were 
donated  to  the  project  by  Schneider  Electric  and  were  estimated  to  be  worth  $30,000.  Because 
each  data  center  is  different,  the  costs  of  piping  and  cooling  infrastructure  should  be  estimated  by 
a  mechanical  contractor  on  a  case-by  case  basis. 

The  costs  to  deploy  this  technology  for  a  given  amount  of  IT  performance  is  likely  to  be  more 
expensive  than  a  conventional  system,  due  to  the  unique  bath  enclosure  and  the  liquid  costs 
associated  with  filling  the  bath.  There  will  be  some  additional  costs  if  a  Novec  649  vapor  level 
monitoring  system  is  required.  The  demonstration  used  a  new  Bacharach  HGM-MZ-4  multi¬ 
channel  vapor  sensing  monitor  costing  $5,845. 
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3M  Novec  649  Liquid 

The  Novec  649  immersion  liquid  is  at  the  heart  of  the  two-phase  cooling  technology.  This 
engineered  low-temperature  boiling  point  liquid  costs  $75  per  liter  (~$290  per  gallon)  at  the  time 
of  this  report.  The  costs  of  the  liquid  is  split  into  two  categories:  initial  fill  and  operational  losses. 

Initial  Fill 

The  approximate  amount  of  Novec  649  liquid  for  the  initial  fill  was  6  “barrels.”  Each 
barrel  is  filled  with  99  liters  (at  20°C).  Therefore  the  cost  of  the  initial  fill  is  estimated  to 
be  $44,550.  It  is  expected  that  the  amount  of  Novec  649  needed  for  the  initial  fill  will  be 
reduced  when  the  IT  hardware  design  has  been  optimized  for  two-phase  immersion 
cooling. 

Operational  Loss 

The  amount  of  Novec  649  lost  during  the  demonstration  was  estimated  to  be  86.94  liters 
with  a  cost  of  $6,520.  This  loss  was  more  than  expected.  Potential  reasons  for  the 
unexpected  loss  are  discussed  in  Section  6.7. 

Bath  and  System  Maintenance 

There  are  a  number  of  systems  and  components  that  were  part  of  the  immersion  bath  that  was  not 
present  in  the  Base  Case  support  systems  and  not  found  on  conventional  computer  racks  housing 
air-cooled  IT  equipment.  These  unique  systems  are  described  in  Section  5.3,  under  Immersion 
Tank  Systems  and  Components.  A  number  of  these  systems  and  components  need  periodic 
monitoring  and  maintenance.  Because  the  demonstration  did  not  proceed  as  planned,  data  on 
regular  periodic  part  replacement  is  not  available.  A  couple  of  the  systems  and  components 
would  need  periodic,  possibly  monthly,  maintenance.  These  systems  are  described  below. 

Humidity  Control 

The  humidity  control  system  (labeled  “7:  desiccant”)  in  Figure  5-6  has  a  cartridge  that 
would  need  replacement  periodically.  The  cost  of  each  cartridge  is  estimated  to  be  $100. 
In  addition,  the  labor  cost  for  monitoring  and  changing  the  cartridge  may  need  to  be 
considered. 

Liquid  Filtering  and  Conditioning 

A  carbon  filter  with  a  circulation  pump  was  included  to  remove  debris  that  might  be 
created  in  the  bath  or  introduced  from  the  outside. 

Perfluoropropionic  acid  (PFPA)  can  form  when  water  comes  in  contact  with  Novec  649. 
Tests  performed  by  3M  during  the  demonstration  indicated  that  controlling  the  acid  levels 
in  Novec  649  is  more  important  than  originally  thought.  Since  the  acid  levels  can  be 


71 


controlled  using  a  filter  containing  silica  gel,  such  a  filter  was  added.  Both  the  carbon  and 
silica  gel  filters  need  to  be  changed  periodically. 

Hardware  Failures 

The  bath  containment  structure  was  a  welded  stainless  steel  tank.  Other  than  a  hidden  defect  in 
the  weldment,  the  basic  structure  should  last  indefinitely.  No  failures  of  the  bath  structure 
occurred  during  the  demonstration  project.  However,  there  were  two  failures  of  the  glass 
top  cover  likely  caused  by  uneven  tightening  of  the  fasteners. 

There  are  a  number  of  components  that  would  need  replacement  over  time.  These  components 
include  fans  (humidity  control,  vapor  recovery,  and  auxiliary  cooling  systems),  one  or  two 
pumps  for  the  liquid  conditioning  and  acid  neutralizing  system  (if  equipped),  and  modulated  and 
passive  valves  (main  condenser  water  flow  control,  vent-in  and  vent-out  systems,  and  lid  seal 
inflation  control).  Servicing  or  replacing  these  components  should  not  be  difficult  for  data  center 
technicians  if  they  are  provided  with  the  needed  training  and  materials.  Although  an  immersion 
bath  has  more  components  than  a  standard  data  center  rack,  there  is  no  reason  for  a  limited 
overall  life,  because  the  components  that  wear  overtime  are  replaceable.  Failures  of  fans,  valves, 
or  seal  system  were  not  observed  during  the  demonstration.  IT  hardware  is  typically  replaced 
every  three  to  five  years  with  models  that  have  higher  performance.  If  the  new  IT  hardware  is  not 
compatible  with  an  existing  immersion  bath  system,  a  different  bath  would  be  needed. 

Operator  Training 

The  demonstrated  immersion  cooling  equipment  has  systems  that  are  part  of  each  tank 
(Figure  5-4).  Data  center  technicians  will  not  initially  be  familiar  with  these  components  and 
systems.  It  is  expected  that  the  technicians  require  a  considerable  amount  of  training  on  the 
theory  of  operation,  electronic  or  physical  monitoring,  and  maintenance  for  the  systems  that  need 
periodic  adjustment  or  are  prone  to  failures.  In  addition,  training  will  be  required  for  procedures 
involving  the  removal  and  reinstallation  of  FT  equipment  because  this  involves  operation  of  the 
lid  sealing  system  and  auxiliary  condenser  system.  The  demonstration  did  not  reach  the  mode  of 
normal  production  and  therefore  data  on  training  hours  are  not  available.  The  research  team 
estimates  that  the  initial  amount  of  training  would  be  one  to  two  weeks  ($3,000)  with  a  few-day 
refresher  course  yearly  ($400).  One  technician  can  care  for  multiple  baths,  but  the  exact  number 
was  not  established. 

7.2  COST  DRIVERS 

There  are  a  number  of  cost  considerations  that  should  be  evaluated  when  comparing  the 
immersion  cooling  technology  with  Novec  649  liquid  to  other  cooling  technologies.  These 
considerations  can  be  divided  into  two  categories:  first  costs  and  operational  costs. 
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First  Costs 


Bath  Structure,  Liquid,  and  Attached  Systems 

The  structure  directly  containing  the  IT  equipment  on  the  data  center  floor  has  a  number 
of  subsystems  and  special  requirements  that  make  the  total  containment  system 
considerably  more  expensive  than  a  typical  data  center  vertical  rack  and  more  expensive 
than  the  systems  that  come  with  the  Base  Case  configuration. 

The  bath  needs  to  be  filled  with  Novec  649  to  completely  immerse  the  IT  equipment.  The 
liquid  is  much  more  expensive  ($75/liter)  compared  to  cooling  liquids  typically  found  in 
data  centers,  as  mentioned  previously  (air  is  free).  The  demonstration  bath  initial  fill 
volume  was  approximately  595  liters.  The  retail  cost  at  the  time  of  this  report  was 
$75/liter  and  the  cost  of  the  initial  fill  was  approximately  $44,625.  Lower  operational 
costs  may  mitigate  the  high  cost  of  the  coolant. 

Floor  Space  Density 

The  maximum  power  density  of  IT  equipment  per  unit  floor  area  inside  a  data  center  may 
be  considerably  different  for  the  demonstrated  immersion  cooling  solution  compared  to 
other  cooling  technologies.  One  key  limiting  factor  for  immersion  cooling  is  the  fact  that 
the  baths  are  horizontal  rather  than  vertical.  A  horizontal  orientation  will  disqualify  a 
significant  volume  of  the  data  center. 

Operational  Costs 

Lower  Cooling  System  Energy  Use 

An  immersion  cooling  implementation  can  be  used  in  a  new  data  center  or  as  part  of 
upgrading  or  retrofitting  an  existing  data  center.  The  temperature  of  the  water  needed  to 
support  immersion  cooling  is  typically  much  higher  than  that  needed  for  cooling 
infrastructures  commonly  found  in  DoD  data  centers.  The  cooling  water  can  be  produced 
by  using  very  efficient  processes  such  as  dry  cooling  or  water-side  economization, 
thereby  avoiding  the  first  costs  associated  with  installing  or  adding  compressor-based 
cooling  capacity. 

The  demonstrated  two-phase  cooling  technology  can  provide  significant  cooling  energy 
savings,  depending  on  the  cooling  infrastructure  design  and  climate.  The  overall  energy 
savings  obtained  from  modeling  the  energy  use  was  19  percent  compared  to  the  Base 
Case.  More  information  is  available  in  Section  6.2. 


Liquid  Loss 
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During  the  operating  period  from  October  2014  to  March  2015,  a  significant  amount  of 
liquid  was  unaccounted  for.  Even  though  the  headspace  (Figure  2-1)  contains  a  layer  of 
air  at  the  top,  apparently  a  significant  amount  of  Novec  649  vapor  escapes  through  the  lid 
seal  or  over  the  edge  of  the  tank  during  periods  when  the  lid  is  removed  The  mechanisms 
of  liquid  and  vapor  loss  were  not  investigated  in  detail.  Future  research  should  investigate 
and  quantify  Novec  649  liquid  loss  after  the  tank  is  initially  filled  and  during  normal 
operation  and  maintenance  of  IT  equipment.  The  cost  of  Novec  649  liquid  lost  during  the 
demonstration  was  significant  compared  to  the  cost  of  the  electrical  energy  needed  for 
operating  the  IT  equipment  (a  metric  used  for  quantifying  Performance  Objective  7). 
Fiquid  loss  cost  analysis  details  are  included  as  part  of  evaluating  this  performance 
objective  in  Section  6.7  above. 

Tank  Systems  Maintenance 

There  were  a  number  of  systems  included  with  the  bath  in  this  demonstration  (Figure  5-4, 
Section  5.2.3).  Some  of  these  systems,  such  as  the  venting  and  vapor  recovery,  might  be 
candidates  for  centralization,  thereby  reducing  both  first  and  operational  costs.  However, 
other  systems  may  not  be  able  to  be  centralized;  for  example,  moisture  control  (desiccant 
filter)  and  liquid  contamination  control  (carbon  or  silica  gel  filter).  During  the  project  we 
did  not  encounter  failures  of  any  of  these  systems. 

Systems  associated  with  the  Base  Case  configuration  contained  fans,  pumps,  and  other 
devices  that  would  also  eventually  need  maintenance.  There  was  little  evidence,  other 
than  the  cracked  glass  lid,  that  the  immersion  bath  and  its  systems  would  require  more 
maintenance  than  the  Base  Case. 

7.3  COST  ANALYSIS  AND  COMPARISON 

A  seven-year  life  cycle  cost  comparison  was  made  using  the  NIST  building  life  cycle  cost 
calculator  as  part  of  obtaining  results  for  Performance  Objective  8,  System  Economics.  Results 
and  additional  information  are  provided  in  Section  6.8. 

The  cost  comparison  required  a  decision  between  replacing  IT  equipment  ready  for  retirement 
with  IT  equipment  designed  for  Base  Case  cooling  technology  or  IT  equipment  designed  for  OBI 
immersion  cooling. 

Many  of  the  costs  used  for  the  life  cycle  analysis  were  based  on  costs  incurred  or  results  obtained 
as  part  of  the  demonstration.  Continued  bath  design  development  and  materials  obtained  in 
production  quantities  should  substantially  lower  the  first  costs  and  operational  cost  (liquid  loss). 
There  were  no  attempts  to  estimate  future  costs. 
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8.0  IMPLEMENTATION  ISSUES 


8.1  REGULATIONS 

3M  Novec  649  liquid  has  a  very  low  GWP  of  1,  which  is  the  same  as  carbon  dioxide  (CO2). 
Novec  649  liquid  is  also  sold  as  Novec  1230,  which  is  a  fire  suppression  chemical  used  in  many 
data  centers.  There  are  no  known  regulations,  special  permits,  or  air-shipment  restrictions 
associated  with  either  Novec  649  or  Novec  1230. 

8.2  END-USER  CONCERNS  AND  DECISION-MAKING  FACTORS 

The  Naval  Research  Laboratory  end  user  had  a  number  of  safety  and  operational  concerns.  These 
concerns  are  listed  in  the  details  for  Performance  Objective  10  in  Section  6.10. 

8.3  ELECTRONICS  FAILURES 

Failures  of  immersion-cooled  electronics  were  the  primary  operational  concern.  The  failures 
were  divided  into  two  groups:  (1)  power  supplies  and  (2)  logic  boards.  The  failures  occurred  in 
both  groups  after  short  periods  of  operation  in  the  immersion  cooling  bath. 

Power  supply  failures  were  caused  by  shorting  across  the  leads  of  certain  field  effect  transistors 
(FETs)  (Figure  8-1). 


Figure  8-1:  Failed  Power  Supply  FET  (courtesy  of  Delta  Electronics) 


The  power  supply  failures  appeared  to  be  caused,  in  part,  by  accumulations  of  contaminants 
absorbed  into  the  Novec  649  liquid  from  the  immersed  electronic  equipment,  namely  the  oils 
contained  in  wire  or  wire  harness  sleeving.  These  accumulations  (Figure  8-2),  which  by 
themselves  may  not  be  conductive,  may  form  a  “base”  structure  where  electrically  conductive 
debris  (for  example,  “whiskers”)  or  chemicals  accumulate,  forming  electrically  conductive  paths 
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resulting  in  component  or  electronic  assembly  failure. 


Figure  8-2:  “Goop”  With  Whisker  (courtesy  of  3M) 

The  whisker-like  objects  were  also  observed  (Figure  8-3)  in  and  near  the  structures  assumed  to 
be  the  cause  of  shorting. 

Element  analysis  of  these  objects  resulted  in  finding  them  to  be  composed  primarily  of  metals, 
including  tin  (Sn)  (26  percent)  and  Aluminum  (Al)  (42  percent). 
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Figure  8-3:  Metallic  Whiskers  (courtesy  3M) 

Repair  and  prevention  treatment  processes  were  developed,  and  failed  power  supplies  were 
repaired  and  treated.  All  remaining  power  supplies  received  the  preventative  treatment  and  were 
returned  to  the  immersion  cooling  demonstration.  Subsequent  power  supply  failures  were  not 
observed. 

The  power  supply  repair  and  treatment  process  started  with  removing  damaged  parts,  cleaning 
the  area,  installing  new  parts  with  any  encapsulation  (around  the  component  pins)  removed,  and 
applying  a  sealant  to  prevent  Novec  649  from  coming  into  contact  with  the  hot  components  in 
confined  areas.  Encapsulation  was  removed  to  avoid  creating  hidden  boiling  locations  under  the 
encapsulation.  In  addition,  a  boiling  enhancement  coating  was  applied  to  large,  flat  surfaces  of 
hot  components  to  encourage  boiling  where  contamination  could  not  accumulate. 

The  logic  board  failures  occurred  later  in  the  demonstration  compared  to  the  power  supply 
failures.  The  apparent  cause  or  causes  of  these  failures  were  not  as  apparent  compared  to  the 
cause  of  the  power  supply  failures.  Metallic  whiskers  assumed  to  collect  under  electronic 
components  that  have  a  high  number  of  contact  points  in  common  with  the  printed  wiring  board. 

The  failures  could  not  be  traced  to  an  exact  root  cause.  The  current  understanding  is  that  a 
combination  of  contamination,  including  “goop”,  acid,  and  whiskers,  caused  undesirable 
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electrical  connections  resulting  in  failures.  The  failures  resulted  in  permanent  component  damage 
(restart  attempts  were  unsuccessful)  for  58  of  the  72  nodes.  Failed  boards  were  replaced  until  the 
spares  were  exhausted.  Fourteen  (14)  logic  boards  (nodes)  continued  to  operate  throughout  the 
demonstration  period. 

8.4  PERFLUOROISOBUTYLENE  (PFIB)  EXPOSURE 

The  SDS  for  Novec  649  states,  “If  the  product  is  exposed  to  extreme  condition  of  heat  from 
misuse  or  equipment  failure,  toxic  decomposition  products  that  include  hydrogen  fluoride  and 
perfluoroisobutylene  (PFIB)  can  occur.”  (SDS  2015a). 

As  part  of  the  demonstration,  a  worst-case  analysis  was  performed  to  estimate  the  amount  of 
PFIB  vapor  that  may  be  formed  if  an  electric  arc  event  occurred.  A  3M  analysis  concluded  that 
there  was  very  little  danger  of  significant  PFIB  exposure  during  the  demonstration.  However,  it 
is  recommended  that  site  safety  personnel  consult  with  appropriate  experts  to  evaluate  the 
potential  risk  in  each  and  every  application. 

8.5  LIQUID  LOSS 

The  quantity  of  Novec  649  liquid  lost  during  the  demonstration  was  significant,  as  was  reported 
in  Section  3  under  Performance  Objective  7.  Losing  the  expensive  liquid  was  a  serious  cost 
concern.  Additionally,  an  inventory  of  make-up  liquid  was  required  at  all  times  to  prevent 
automatic  shutdown  of  the  electronic  equipment.  The  demonstration  bath  had  a  Novec  649  liquid 
level  sensor  that  was  programmed  to  cut  the  power  to  the  immersion-cooled  electronics  if  the 
liquid  level  fell  below  the  highest  point  of  the  equipment.  The  cause  or  causes  of  the  liquid  loss 
should  be  understood  and  addressed  to  reduce  the  loss  to  an  acceptable  rate. 

8.6  SERVICE  VISIBILITY 

The  boiling  of  Novec  649  can  produce  a  significant  amount  of  vapor  bubbles,  especially  when 
the  IT  equipment  power  level  is  high.  Indicator  lights  are  often  part  of  electronic  equipment 
designs.  Personnel  at  NRL  reported  that  the  poor  visibility  caused  by  the  bubbles  was  hindering 
equipment  monitoring,  including  the  inability  to  read  the  indicator  lights.  FT  equipment  designed 
specifically  for  two-phase  immersion  cooling  should  have  the  indicator  lights  located  so  that  they 
are  always  visible  to  the  service  personnel. 
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8.7  USE  OF  GLOVES  FOR  SERVICE 


Personnel  at  NRL  reported  that  using  rubber  gloves  while  reaching  into  the  liquid  to  perform 
service  work  was  desirable  from  a  perceived  safety  and  comfort  point  of  view  compared  to  using 
bare  hands.  Dexterity  required  for  certain  service  work  was  reported  to  be  difficult  when  using 
the  gloves.  Personnel  elected  to  perform  service  without  gloves  although  it  dried  out  their  skin. 
However,  this  is  not  a  recommended  practice. 

8.8  LID  DURABILITY 

The  removable  glass  lid  covering  the  top  of  the  bath  was  difficult  to  handle  because  of  its  size 
and  weight.  The  lid  cracked  twice  during  the  demonstration.  Future  bath  enclosures  should  use 
an  improved  design  that  address  these  issues. 

8.9  PROCUREMENT  ISSUES 

The  bath  enclosure  design  and  fabrication  methods  were  not  mature  enough  for  high-volume 
production,  making  the  bath  enclosures  expensive  and  delivery  times  unpredictable. 
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APPENDICES 

Appendix  A:  Points  of  Contact 
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katelyn.rydberg  @navy.mil 
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Vem  Novstrup 

Naval  Facilities 
Engineering  and 
Expeditionary  W  arfare 
Center 

vern.  novstrup  @  navy.mil 

Technical  Advisor 

Dr.  Magnus  Herrlin 

Lawrence  Berkeley 
National  Laboratory 

mkherrlin@lbl.gov 

Principal  Investigator 

BillTschudi 

Lawrence  Berkeley 
National  Laboratory 

Retired 

Principal  Investigator 

Henry  Coles 

Lawrence  Berkeley 
National  Laboratory 
Address* 

hccoles@lbl.gov 

Technical  Lead 

Dr.  Jeanie  Osburn 

Naval  Research 
Laboratory 

Retired 

Project  Lead 

Dr.  Sergio  Tafnr 

Naval  Research 
Laboratory 

sergio .  tafur  @  nrl.  navy .  mil 

Project  Lead 

Phil  Tuma 

3M  Corporation 

petuma@3M.com 

Project  Lead 

OzanTutunoglu 

Schneider-Electric 

ozan.tutunoglu  @  schneider- 
electric.com 

Project  Lead 

John  Bean 

Schneider-Electric 

j  ohn.bean  @  sc  hneider- 
electric.com 

Management  Support 

Tim  McCann 

SGI 

tmccann@sgi.com 

Project  Lead 
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Appendix  B:  Instrument  Calibration 


Liquid  Level  Sensor 

The  liquid  level  was  used  to  determine  the  Novec  649  liquid  loss  during  the  evaluation  period. 
The  instrument  was  a  magnetic-switch  type  model  Omega  LVR-524  Liquid  Level  Sensor.  The 
translated  output  (Figure  6.7-1,  top  graph)  from  the  sensor  was  recorded  in  the  database. 

The  correspondence  between  the  database  values  (counts)  and  the  actual  liquid  level  (inches) 
was  confirmed,  and  the  liquid  volume  changes  as  a  function  of  liquid  level  were  estimated. 

Magnetic-switch  type  level  sensors  do  not  provide  a  continuous  output  as  a  function  of  float 
height.  The  output  is  a  series  of  “steps”  corresponding  to  the  activation  of  a  linear  arrangement 
of  magnetic  switches.  These  steps  are  seen  in  the  top  graph  of  Figure  6.7-1. 

The  documentation  for  the  Omega  LVR-524  indicated  that  the  active  range  for  the  sensor  is 
21.37  inches  (542.8  millimeters  [mm]).  The  sensor  output  was  4-20  milliamps  [mA],  This  output 
was  translated  linearly  to  a  value  between  0-4096  (referred  to  as  a  count)  and  placed  in  the 
database.  The  translation  (database  value  to  liquid  height  change)  based  on  documentation  was 
therefore  0.005217  inches  (0.1325  mm)  for  a  single  count,  or  191.67  counts  per  inch. 

The  process  described  below  was  performed  to  confirm  the  translation  (191.67  counts/inch). 
Confirming  the  count/inch  was  accomplished  in  three  steps.  Given  that  if  the  count/liter  and 
liters/inch  can  be  determined  experimentally,  the  count/inch  and  other  relationships  (Table  B-2) 
can  be  estimated  using  equation  B-l. 

(liters/inch)  x  (count  change/liter)  =  count  change/inch  (Eq.  B-l) 

Step  1:  Experimentally  estimate  the  bath  liters  per  height  change.  Measure  the  tank  dimensions 
to  estimate  the  liquid  surface  area.  Two  NRL  technicians  provided  measurements  that  estimated 
the  area  of  the  liquid  surface.  The  data  and  results  (effective  surface  area  =  0.871  m2)  are  in 
Table  B-l . 

Step  2:  Experimentally  estimate  the  count  change  as  a  function  of  liquid  added.  A  known  volume 
of  liquid  was  added  to  the  bath  in  three  consecutive  and  equal  amounts  (10.64  liters)  while  the  IT 
equipment  power  was  off  and  the  liquid  was  at  room  temperature.  The  change  in  database  values 
were  observed.  Table  B-l  shows  the  data  and  results. 

Step  3:  Combine  results  to  obtain  an  estimate  of  count  change  per  height  change.  Combine  the 
information  from  Step  1  and  2.  Liters  per  inch  (22.12)  multiplied  by  counts  per  inch  (8.68)  =  192 
(count/inch). 
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The  experimental  results  (192  counts/inch)  are  close  to  the  instrument  documentation 
information  (191.7  counts/inch).  The  value  of  192  counts/inch  was  used  for  converting  the  level 
measurements  into  liquid  volume  changes. 

The  resulting  estimate  of  8.68  count/liter  (Table  B-2)  was  used  to  calculate  the  volume  of  liquid 
needed  to  raise  the  liquid  level  from  that  found  at  the  end  of  the  liquid  loss  evaluation  period  to 
the  level  found  at  the  beginning  of  the  period. 


Table  B-l:  Level  Sensor  Translation  Experimental  Data  And  Results 


Estimated  Width 
(inches) 

Estimated  Length 
(inches) 

Calculated  Area 
(inchesA2) 

Estimated  Area 
(mA2) 

Volume/Height 

(mA3/m) 

Volume/Height 

(liters/mm) 

Volume/Height 

(liters/inch) 

Tech  1 

39 

32 

1248.0 

0.805 

0.805 

0.805 

20.45 

Tech  2 

39.25 

37 

1452.3 

0.937 

0.937 

0.937 

23.80 

Average 

39.125 

34.5 

1350.1 

0.871 

0.871 

0.871 

22.12 

Date/Time 

Data 

(count) 

Data 

(count) 

Data  Delta 
(count) 

Observed  Rate 
(count/liter) 

Add  10.64  Liters 

9/2/2015  14:03 

1649.7 

1728 

78.3 

7.36 

Add  10.64  Liters 

9/2/2015  15:00 

1728 

1828 

100 

9.40 

Add  10.64  Liters 

9/2/2015  16:16 

1828 

1926.7 

98.7 

9.28 

Average 

92.33 

8.68 

Calculate 

(count/inch) 

192.00 


Table  B-2:  Liquid  Level  Related  Conversions 


Fluid  Surface 

Area 

0.871 

mA2 

Level  Sensor 

Data 

Conversion 

192 

count/inch 

Conversions 

7559 

count/m 

7.56 

count/mm 

0.1323 

mm/count 

7559 

count/871  liters 

8.68 

count/liter 

871 

liters/meter 

0.871 

liters/mm 

22.12 

liters/inch 
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Appendix  C:  Simulation  Model  and  Simulation  Details 


Model  Development 

Two  types  of  data  center  models  were  developed:  One  for  a  low-efficiency  data  center  (used  to 
model  Base  Case  A  and  Immersion  Cases  1A,  2A,  and  3A),  and  one  for  a  high-efficiency  data 
center  (used  to  model  Base  Case  B  and  Immersion  Cases  IB,  2B,  and  3B).  All  these  models  use 
the  same  IT  equipment.  The  differences  are  listed  in  Table  C-l. 

The  Base  Case  models  (A  and  B)  assume  the  IT  equipment  is  equipped  with  the  commercially 
available  SGI  hybrid-cooling  option. 

The  high-efficiency  data  center  is  equipped  with  high-efficiency  chillers  and  uninterruptible 
power  supply  (UPS)  systems.  In  addition,  the  high-efficiency  data  center  is  equipped  with  a 
water-side  economizer  (WSE)  and  has  excellent  air  management  inside  the  data  hall.  A  water¬ 
side  economizer  allows  the  chiller  to  remain  off  when  the  cooling  water  can  be  supplied  at  the 
required  temperature  using  only  the  cooling  tower.  This  feature  provides  significant  energy 
savings  in  many  climates. 

The  low-efficiency  data  center  is  equipped  with  low-efficiency  chillers  and  UPS  systems  and 
does  not  include  a  WSE.  The  comparisons  for  Performance  Objectives  1,  2,  6,  and  8  used  the 
simulation  results  from  the  low-efficiency  data  center  when  comparing  the  Base  Case  (hybrid 
cooling)  and  Immersion  Cooling.  The  simulation  results  of  the  high-efficiency  data  center 
contrasted  with  the  simulation  results  of  the  low-efficiency  data  center.  The  PUE  results  using  a 
high-efficiency  data  center  were  1.216  for  Base  Case  B  and  1.146  for  Immersion  Case  3B.  These 
data  result  in  a  6  percent  savings  on  the  total  data  center  energy  compared  to  19  percent  for  the 
low-efficiency  data  center. 


Table  C-l:  Low-  and  High-Efficiency  Data  Center  Differences 


Data  Center  Efficiency 

Low 

High 

Cooling  Tower/Chiller 
Configuration 

no  water-side  economizer 

water-side  economizer 

Chiller 

RC  Group  W.NRM.2760  V4 

McQuay  DWSC100M 

UPS 

MGE  Galaxy  7000 

Symmetra  MW 

Both  the  low-  and  high-efficiency  data  centers  have  a  cooling  tower  that  supplies  cool  water  to 
chillers  for  the  whole  data  center  (see  Table  C-l).  However,  for  the  heat  rejection  for  immersion 
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simulations  A3  and  B3  (Table  6.1-1)  a  dedicated  cooling  tower  was  added  to  cool  the  immersion 
baths. 

For  each  of  the  two  base  models  (high  and  low  efficiency),  three  scenarios  were  modeled  to 
explore  the  energy  use  if  the  same  Base  Case  IT  equipment  was  cooled  using  the  demonstrated 
immersion  cooling  technology.  The  first  scenario  sets  (1A  and  IB)  assume  the  immersion 
cooling  bath  condensers  are  connected  to  the  existing  chilled-water  cooling  loop.  Many  HPC 
data  centers  have  cooling  water  available  overhead  or  under  the  raised  floor.  Therefore,  this 
option  has  a  relatively  low  cost. 

The  second  scenario  set  (2A  and  2B)  assumes  that  dry  coolers  with  an  adiabatic  option  are  added 
outside  the  data  center  hall  to  supply  cooling  water  to  the  immersion  bath  condensers.  The 
adiabatic  function  (water  sprayed  on  heat  exchanger  surfaces)  is  used  for  very  hot  ambient 
conditions. 

The  third  scenario  set  (3A  and  3B)  assumes  that  a  cooling  tower  is  added  outside  of  the  data 
center  to  supply  the  cooling  water  to  the  immersion  baths.  Cooling  towers  can  economically 
supply  a  large  amount  of  cooling  in  a  small  space  but  need  a  considerable  amount  of  water  for 
evaporation.  The  cooling  tower  option  may  not  be  a  good  choice  in  a  location  that  has  high  water 
cost  or  shortages. 

The  model  used  a  data  center  designed  for  a  maximum  IT  load  of  2  MW.  The  actual  load  used  in 
the  model  assumed  that  the  IT  load  was  80  percent  of  the  maximum.  The  model  assumed  that 
20  percent  of  the  IT  equipment  was  air  cooled  and  not  be  part  of  the  retrofit.  The  power  for  the 
air-cooled  IT  equipment  that  stays  in  place  was  therefore  321  kW,  with  1,275  kW  for  the  IT 
equipment  being  exchanged  during  the  retrofit. 

Performance  Objectives  1,  2,  6,  and  8  were  evaluated  by  using  two  of  the  scenarios  above  only: 

•  Base  Case  A:  SGI  ICE  X  Model  IT  equipment  equipped  with  the  SGI  hybrid  cooling 
option  installed  in  a  low-efficiency  data  center 

•  Immersion  Case  3A:  The  same  SGI  ICE  X  Model  IT  equipment  modeled  in  the  Base 
Case  A,  but  cooled  using  the  demonstrated  two-phase  immersion  cooling.  A  cooling 
tower  was  added  to  provide  cooling  for  all  immersed  IT  equipment.  This  scenario  was 
also  modeled  in  the  low-efficiency  data  center. 

Eight  Romonet  models  were  developed,  and  the  following  eight  Romonet  software-generated 
energy  flow  diagrams  are  provided  below  (Figures  C-l  through  C-8).  The  red  lines  indicate  heat 
energy  flow.  The  blue  lines  indicate  electrical  energy  flow.  The  boxes  indicate  enclosed 
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boundaries  or  rooms.  The  net  energy  flow  into  and  out  of  each  boundary  is  zero. 


Figure  C-l:  Low-Efficiency  Data  Center  -  Base  Case  A 


-< 


Figure  C-2:  Low-Efficiency  Data  Center  -  Immersion  (1A)  Cooling 
Uses  Building  Cooling  Water 
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Figure  C-3:  Low-Efficiency  Data  Center  -  Immersion  (2A)  Cooling  with  Added  Dry  Cooler 


Figure  C-4:  Low-Efficiency  Data  Center  -  Immersion  (3A)  Cooling 
with  Added  Cooling  Tower 
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Figure  C-5:  High-Efficiency  Data  Center  -  Base  Case  B 


Figure  C-6:  High-Efficiency  Data  Center  -  Immersion  (IB)  Cooling 
with  Building  Cooling  Water 
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Figure  C-7:  High-Efficiency  Data  Center  -  Immersion  (2B)  Cooling  with  Added  Dry  Cooler 


Figure  C-8:  High-Efficiency  Data  Center  -  Immersion  (3B)  Cooling 
with  Added  Cooling  Tower 
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Table  C-2:  Modeling  Results  by  Component 


Low  Efficiency  Data  Center 

High  Efficiency 

Data  Center 

Basecase  A 

Immersion  1A 

Immersion 

2A 

Immersion 

3A 

Basecase  B 

Immersion 

IB 

Immersion 

2B 

Immersion 

3B 

CUD 

Mech  Switchboard 

0.876 

0.812 

0.541 

0.536 

0.560 

0.541 

0.517 

0.510 

c 

UPS  LV  CRAC 

0.432 

0.430 

0.424 

0.424 

0.241 

0.241 

0.241 

0.241 

o 

o 

SGI  CDUPump 

21.7 

21.7 

u 

Data  Hall  CRAC  PDU 

0.185 

0.085 

0.085 

0.085 

0.185 

0.085 

0.085 

0.085 

<D 

~o 

Data  Hall  ITCRAH 

19.61 

13.63 

13.63 

13.63 

19.62 

13.63 

13.63 

13.63 

to 

Immersion  Controls 

5.315 

5.315 

5.315 

5.315 

5.315 

5.315 

SGI  Cooling  Rack 

24.1 

24.1 

Aircooled  IT 

320.9 

320.9 

320.9 

320.9 

320.9 

320.9 

320.9 

320.9 

b : 

Immersed  IT 

1275 

1275 

1275 

1275 

1275 

1275 

Direct  water  cooled  IT 

1275 

1275 

UPS  LV  Switchgear 

2.135 

2.135 

2.135 

2.135 

2.109 

2.109 

2.109 

2.109 

UPS 

25.73 

25.73 

25.73 

25.73 

11.65 

11.65 

11.65 

11.65 

4-» 

u 

LV  Distribution  Cabling 

0.943 

0.943 

0.943 

0.943 

0.943 

0.943 

0.943 

0.943 

Distribution  Cabling 

4.964 

4.964 

4.964 

4.964 

4.964 

4.964 

4.964 

4.964 

to 

Lighting 

50.14 

50.14 

50.14 

50.14 

50.14 

50.14 

50.14 

50.14 

Generator  Preheat 

50.14 

50.14 

50.14 

50.14 

50.14 

50.14 

50.14 

50.14 

Main  Transformer 

21.47 

20.80 

17.08 

16.99 

17.19 

16.79 

16.36 

16.21 

<u 

PDU  to  Aircooled  IT 

0.626 

0.626 

0.626 

0.626 

0.626 

0.626 

0.626 

0.626 

n 

PDU  to  Direct  Cooled  IT 

2.333 

2.333 

PDU  to  Immersion  Cooled  IT 

2.333 

2.333 

2.333 

2.333 

2.333 

2.333 

Chiller 

477.0 

463.5 

92.31 

92.31 

53.72 

56.05 

20.22 

14.76 

Cooling  Tower  Pump 

38.90 

38.30 

13.66 

13.66 

34.86 

34.42 

12.88 

12.84 

Cooling  Tower  Summer 

7.077 

6.643 

1.539 

2.109 

2.584 

2.563 

0.725 

1.311 

c 

Cooling  Tower  Winter 

12.127 

12.085 

3.408 

3.408 

o 

Cooling  Tower 

0.957 

O 

Pump 

34.198 

33.667 

12.718 

12.718 

34.052 

33.493 

12.635 

12.635 

<L> 

"O 

Mech  PDU 

1.105 

1.150 

0.393 

0.381 

0.409 

0.409 

0.350 

0.336 

'to 

Dry  Cooler  pump 

28.639 

28.639 

Dry  cooler 

12.526 

12.526 

Immersion  cooled  pumps 

28.639 

Added  Cooling  Tower 

0.957 

Added  Cooling  Tower  Pump 

28.639 

Total  Power 

2380 

2318 

1932 

1921 

1941 

1895 

1847 

1830 

Total  Cooling  Power 

625 

563 

182 

171 

204 

159 

111 

95 

Total  IT  Power 

1596 

1596 

15% 

1596 

1596 

1596 

1596 

15% 

Cooling  pPUE 

1.392 

1.353 

1.114 

1.107 

1.128 

1.099 

1.070 

1.059 

PUE 

1.491 

1.452 

1.210 

1.203 

1.216 

1.187 

1.157 

1.146 

Measurements  from  the  demonstration  indicate  that  very  low  total  cooling  energy  can  be 
achieved,  including  the  infrastructure  overhead  for  controls,  pumping,  and  dry  cooler  fan  power. 
Measurements  were  taken  during  the  period  from  March  13  to  April  8,  2015.  The  resulting 
lowest  measured  pPUE  value  is  just  below  1.02,  shown  on  the  bottom  graph  of  Figure  6.1-3.  In 
Table  C-2,  the  lowest  pPUE  is  1.059,  but  this  cooling  energy  includes  inefficient  cooling  for 
20  percent  of  the  IT  load. 
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Appendix  D:  Base  Case  Test  Measurements 


The  net  heat  loss  to  the  room  during  the  Base  Case  test  was  needed  as  a  model  input  for  Base 
Case  A  and  Base  Case  B  because  this  heat  affects  the  heat  load  to  the  computer  room  air 
handlers  (CRAHs)  Tables  D-l  through  D-3  contain  the  measurements,  as  well  as  the  calculated 
results.  All  results  are  summarized  in  Table  D-4,  including  the  “Unaccounted  Power.” 

The  identifiers,  e.g.,  “PI”  in  Table  D-l,  relate  to  the  sampling  points  indicated  on  Figure  5-5. 

The  equation  for  the  unaccounted  heat  is: 

Unaccounted  Power  =  Power  In  (electrical  energy)  -  Power  Out  (Cooling  Rack  and  CDU) 

(Eq.  D-l) 

Referring  to  Figure  5-5,  the  equation  needed  to  determine  unaccounted  power  is: 

P7  =  (PI  +  P2  +  P3)  -  (P6  +  P5)  (Eq.  D-2) 

Power  In:  PI,  P2,  and  P3  are  measured  directly  and  totaled  in  Table  D-l. 

Power  Out:  P6  is  rack  air  cooling  by  cooling  rack  (Table  D-2). 

Power  Out:  P5  is  rack  liquid  cooling  by  CDU  (Table  D-3). 
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Table  D-l:  Base  Case  Test  Energy  Input 


Time 

Measurements 

Calculated 

[pi] 

Cooling  Rack 
Electrical 
(kW) 

[P2] 

IT 

Equipment 

(kW) 

[P3] 

CDU Pump 

Power 

(kW) 

Electrical 

Power  In 

P1+P2+P3 

(kW) 

11:00 

2.06 

28.5 

3.2 

33.7 

11:04 

28.5 

3.2 

33.8 

11:08 

28.5 

3.2 

33.7 

11:12 

28.4 

3.2 

33.7 

11:16 

2.04 

28.4 

3.2 

33.7 

11:20 

2.04 

28.3 

3.2 

33.6 

11:24 

2.06 

28.4 

3.2 

33.7 

11:28 

2.05 

28.4 

3.2 

33.7 

11:32 

2.05 

28.3 

3.2 

33.5 

11:36 

2.01 

28.3 

3.2 

33.5 

11:40 

2.02 

28.2 

3.2 

33.4 

11:44 

2.00 

28.4 

3.2 

33.6 

11:48 

2.02 

28.8 

3.2 

34.0 

11:52 

2.02 

28.4 

3.2 

33.7 

11:56 

2.03 

28.2 

3.2 

33.5 

12:00 

2.04 

28.4 

3.2 

33.6 

12:04 

2.01 

28.1 

3.3 

33.4 

Average 

2.04 

28.38 

3.23 

33.64 
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Table  D-2  shows  the  measurements  and  calculations  for  determining  the  heat  going  to  the 
building  cooling  system  from  the  Cooling  Rack  (rack  air  cooling). 


Table  D-2:  Base  Case  Cooling  Rack  (Rack  Air  Cooling)  Measurements  and  Results 


Time 

Measurements 

Calculated 

[F2] 
Cooling 
Rack  (CR) 
Flow  Rate 

(gpm) 

[T3] 

CR 

Supply 

Temp. 

(C) 

[T4] 

C2 

Return 

Temp. 

(C) 

[T5] 

Cl 

Return 

Temp. 

(C) 

C2  Heat 

Transfer 

(kW) 

Cl  Heat 

Transfer 

(kW) 

[P6] 

C1+C2 

Heat 

Transfer 

(kW) 

11:00 

24.4 

20.0 

20.0 

21.0 

0.00 

3.22 

3.22 

11:04 

24.3 

20.0 

20.0 

21.0 

0.00 

3.21 

3.21 

11:08 

23.6 

21.0 

20.0 

21.0 

-3.11 

0.00 

-3.11 

11:12 

24.4 

21.0 

20.0 

21.0 

-3.22 

0.00 

-3.22 

11:16 

24.4 

21.0 

20.0 

21.0 

-3.22 

0.00 

-3.22 

24.6 

21.0 

21.0 

22.0 

0.00 

3.24 

3.24 

24.4 

20.0 

21.0 

0.00 

3.22 

3.22 

11:28 

24.8 

20.0 

20.0 

21.0 

0.00 

3.27 

3.27 

11:32 

24.6 

20.0 

20.0 

21.0 

0.00 

3.25 

3.25 

11:36 

24.2 

19.0 

19.0 

21.0 

0.00 

6.39 

6.39 

11:40 

24.4 

19.0 

19.0 

21.0 

0.00 

6.45 

6.45 

11:44 

22.9 

19.0 

20.0 

0.00 

3.02 

3.02 

11:48 

24.5 

19.0 

20.0 

0.00 

3.23 

3.23 

11:52 

24.2 

19.0 

21.0 

0.00 

6.39 

6.39 

11:56 

24.6 

20.0 

21.0 

3.24 

6.48 

9.72 

12:00 

24.5 

20.0 

20.0 

21.0 

0.00 

3.23 

3.23 

12:04 

24.3 

20.0 

20.0 

21.0 

0.00 

3.21 

3.21 

Average 

24.31 

19.88 

19.76 

20.94 

-0.37 

3.40 

3.03 

Note:  Negative  values  for  heat  transfer  may  be  caused  by  the  small  number  of  significant  digits 
available  for  temperature  and/or  calibration  offsets. 
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Table  D-3  shows  the  measurements  and  calculations  for  determining  the  heat  going  to  the 
building  cooling  system  from  the  CDU  (rack  liquid  cooling). 


Table  D-3:  Base  Case  CDU  (Rack  Liquid  Cooling)  Measurements  and  Results 


Measurements 

Calculated 

Measurment 

Calculated 

[FI] 

[Tl] 

[T2] 

[P4] 

[P3] 

[P5] 

CDU 

IT 

IT 

IT  Equipment 

CDU 

CDU 

flow 

Equip. 

Equip. 

Direct 

Pump 

Thermal  to 

rate 

Supply 

Return 

Cooling 

Power 

Building 

Time 

(F) 

(F) 

(kW) 

(kW) 

(kW) 

11:00 

68.9 

68.6 

71.2 

26.2 

3.20 

29.4 

11:04 

68.9 

68.8 

71.4 

26.3 

3.23 

29.5 

11:08 

68.9 

68.8 

71.4 

26.3 

3.23 

29.5 

11:12 

68.9 

68.4 

70.6 

22.2 

3.23 

25.4 

11:16 

68.9 

68.0 

70.6 

26.2 

3.24 

29.5 

11:20 

68.8 

67.6 

70.2 

26.2 

29.5 

11:24 

68.8 

67.2 

28.2 

31.5 

11:28 

68.8 

67.0 

69.6 

26.2 

3.23 

29.5 

11:32 

68.8 

66.8 

69.2 

24.2 

3.22 

27.4 

11:36 

68.8 

66.6 

68.8 

22.2 

3.23 

25.4 

11:40 

68.9 

66.4 

68.8 

24.2 

3.22 

27.4 

11:44 

68.8 

66.2 

68.6 

24.2 

3.22 

27.4 

11:48 

68.8 

66.2 

22.2 

3.22 

25.4 

11:52 

68.8 

66.4 

68.8 

24.2 

3.21 

27.4 

11:56 

68.8 

66.8 

24.2 

3.23 

27.4 

12:00 

68.9 

67.2 

69.4 

22.2 

3.24 

25.4 

12:04 

68.8 

67.4 

70.0 

26.2 

3.25 

29.5 

Average 

68.86 

67.32 

69.78 

24.81 

3.23 

28.04 
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Table  D-4  shows  the  results  from  Tables  D-l  through  D-3  and  the  calculated  unaccounted  power 
that  will  be  cooled  by  the  room  air  conditioning  system. 


Table  D-4:  Base  Case  Unaccounted  Power 


Time 

Calculated 

Electrical 

Power  In 

P1+P2+P3 

(kW) 

[P6] 

C1+C2 

Heat 

Transfer 

(kW) 

[P5] 

CDU 

Thermal  to 
Building 
(kW) 

[P7] 

Unaccounted 

for  Power 
(kW) 

11:00 

33.7 

3.2 

29.4 

l.i 

11:04 

33.8 

3.2 

29.5 

l.i 

11:08 

33.7 

-3.1 

29.5 

7.4 

11:12 

33.7 

-3.2 

25.4 

11.5 

11:16 

33.7 

-3.2 

29.5 

7.4 

11:20 

33.6 

3.2 

29.5 

0.9 

11:24 

33.7 

3.2 

31.5 

-1.0 

11:28 

33.7 

3.3 

29.5 

0.9 

11:32 

33.5 

3.2 

27.4 

2.9 

11:36 

33.5 

6.4 

25.4 

1.7 

11:40 

33.4 

6.4 

27.4 

-0.4 

11:44 

33.6 

3.0 

27.4 

3.2 

11:48 

34.0 

3.2 

25.4 

5.3 

11:52 

33.7 

6.4 

27.4 

-0.2 

11:56 

33.5 

9.7 

27.4 

-3.7 

12:00 

33.6 

3.2 

25.4 

5.0 

12:04 

33.4 

3.2 

29.5 

0.7 

Average 

33.64 

28.04 

2.57 
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